Help: Three identical Pis. One SD card. Only one Pi boots.

[bfinio]

I work for a small nonprofit that developed a Raspberry Pi kit and curriculum based on the Model B+. Our kit includes an SD card with slightly customized version of Raspbian from 2014 (we don't have the manpower to keep up with hardware/software updates as fast as they come out, so this is what I've got for now. Bear with me.). Recently we had two Pis returned to us, along with their respective SD cards, because customers claimed they wouldn't boot up - they would either hang on some error message or the screen would go blank.

Wanting to test this for myself, I assumed the customers just wound up with bad SD cards (which they may have caused themselves by improper shutdowns). I tested both Pis and cards and found what the customers did - the Pis would either start booting then the screen would go blank, or they would hang on some error message (more on those in a minute). So, to confirm that the problem is the cards, I stick them each in a third B+ that I already had in my office...and it boots up just fine. Weird.

So, maybe the problem is actually with the Pis somehow? I flash a new card with the latest Raspbian to try that...and both of the returned Pis boot up fine. So, nothing inherently wrong with them. Now I'm really confused. Time for more tests.

I have three different Pis (all Model B+ V1.2, 2014), two SD card images (our 2014 Raspbian image and the May 2016 Raspbian), and two different physical cards (a Kingston card from Amazon and one of the RPi-branded ones from SparkFun). That gives me twelve different boot combinations depending on which image I flash to which card. Everything else - USB keyboard, power supply, HDMI-DVI cable and monitor always remains the same. I ran through dozens of boots with all combinations and this is what I found:

- Pi #1 (the one I had in my office) always boots, regardless of the image or physical card
- Pis 2 and 3 (the ones that got returned) only boot with the latest Raspbian, regardless of the image or physical card
- Pis 2 and 3 give all sorts of crazy, inconsistent behavior when I try to boot them with the older Raspbian. A list of that behavior, as best as I could document it, is below.

So, at this point, I'm pretty stumped. I can't imagine why, given three supposedly identical models of the Pi, two of them would never successfully boot from an SD card that works fine with a third Pi. If the problem was the SD card itself I would have expected at least one failed boot on the good Pi, but nope. The only other guess I have is that the Pis come from different production runs/factories and maybe have slight variations in the firmware that are causing the errors with older Raspbian? I notice that the SoC on the Pi in my office says "SAMSUNG K4P4G324EQ-RGC2" whereas the returned Pis say "4DAI8 D9QHN" and "3UA78 D9QHN" respectively. Not sure if there's other identifying information on the Pis that could tell me when or where they were manufactured, or if that's a wild goose chase anyway since it shouldn't matter.

Here's a list of the behavior I saw when the Pis failed to boot:

- Starts to boot then screen goes blank, too fast for me to read the last line of text before it goes black
- I get crazy colored vertical stripes on the screen and it freezes
- Text stays on screen but gets distorted with crazy colored, flickering flecks everywhere on screen
- It freezes with the last line of text on the screen as one of the following (I tried googling many of these without much luck)
- [warn] Kernel lacks cgroups of memory controller not available, not starting cgroups … (warning). [27.636199] -- [end trace 7cff435ea161522a ]---
- [29.695409] Fixing recursive fault but reboot is needed! [ ok ] Starting enhanced syslogd: rsyslogd[….] Network Interface Plugging Daemon…skip eth0…done. [info] Initializing cgroups. [….] _
- Kernal panic - not syncing - fatal exception in interrupt
- Unable to handle kernel paging request at virtual address beecb140
- [28.577165] [<c03c22640>] (usbnet_get_link) from [<c0484998>] (dev_ethtool+0x17b4/0x1b84)
- INIT: entering runlevel: 2
- Segmentation fault (crazy yellow flecks on screen)

I know that was a long post...any help would be appreciated!

[FTrevorGowen]

It's possible that the three Pi's are not quite identical - perhaps they have different makes** of memory on board? That's probably best checked by trying an SD card with a current version of Raspbian installed.
Trev.
**IIRC the Pi's "firmware" has been updated to accommodate such changes from time-to-time. Also, IIRC, on a B+ the memory chip is the PoP package (mounted on top of the SoC) by the Raspberry logo - my, quite old, B+'s have Samsung chips.

[pi-anazazi]

- Pis 2 and 3 (the ones that got returned) only boot with the latest Raspbian, regardless of the image or physical card

...sometimes it's better to read the post, before answering

[bfinio]

Circling back to answer my own question...looks like the problem was caused by overclocking. I forgot to mention in the original post that our SD card comes overclocked to Turbo. If I reset overclocking to "none" on a card using the working Pi, then the other two Pis boot fine. If I set those back to Turbo and reboot, they crash again.

So, whatever the slight difference between the Pis is - maybe different makes of memory as Trevor mentioned? - it appears that one of them is more tolerant to overclocking while two are not. I know "so don't overclock" seems like an obvious answer but we decided it was worth the risk at the time, because otherwise the desktop interface was way too sluggish for kids running Scratch and the browser at the same time.

So, stuck with it for now, but at least I know what the problem is!

[DougieLawson]

If they're RPi2Bs or RPI3Bs then get rid of Midori and Epiphany and replace the browser with one that works - Chromium 51.

[bfinio]

If they're RPi2Bs or RPI3Bs

thanks but they are Pi1 B+'s

[DougieLawson]

If they're RPi2Bs or RPI3Bs

thanks but they are Pi1 B+'s

Then replace Midori / Epiphany with gkreidl's kweb or kweb3 (which has had some of the webkit bugs knocked out of it).

[bfinio]

Sorry if this wasn't clear in the original post - this is a kit with a pre-flashed SD card. Changing anything on the card is not an option at this point. I was just trying to reverse-engineer to figure out what happened and it appears that overclocking was the problem. The crash happens during boot, before any browsers get opened. Are you suggesting that uninstalling the existing browsers and using the ones you mentioned would prevent this sort of crash? Or just that those browsers are better options in general?

[davidcoton]

I think Dougie was responding to your comment about a sluggish browser.

In general, if you are distributing kits then you do need to be prepared to update your OS occasionally. hardware (component) changes happen, and sometimes need an OS change. You can expect new OS versions to work on older kit, but not necessarily new kit to work with older OSs.

I appreciate that you can't keep up with all OS changes, but perhaps you should check your pre-built card on a ample Pi each time you buy more hardware?

[bfinio]

It's a long story as to why we don't update more frequently so I'll spare that explanation here. You're right though, I checked and we didn't have any problems with the first batch of our kits, this started happening with a re-order of a couple hundred units. The failure rate is low (under 3%) so spot-checking a few kits in each new batch could be very hit or miss. Ideally, one day we'll update to the Pi 3 and change our model for how we do the SD cards, and it won't be an issue...but like I said, this is what I've got to work with for now.

[davidcoton]

It's a long story as to why we don't update more frequently so I'll spare that explanation here. You're right though, I checked and we didn't have any problems with the first batch of our kits, this started happening with a re-order of a couple hundred units. The failure rate is low (under 3%) so spot-checking a few kits in each new batch could be very hit or miss. Ideally, one day we'll update to the Pi 3 and change our model for how we do the SD cards, and it won't be an issue...but like I said, this is what I've got to work with for now.

That, of course, is what you would expect with overclocking -- a few random failures, but enough to be annoying. Random testing won't detect them reliably, but will tell you when an OS change is required by the hardware (expect 100% failure rate).

I believe all new Pis (A A+ B B+ Z) should now work at 1GHz.