stephanst
Posts: 1
Joined: Mon Jul 09, 2018 1:36 pm

Netboot issues with mod 3 B+

Mon Jul 09, 2018 1:49 pm

Hi guys,

It seems there is a problem with the default bootcode on the Raspberry Pi 3 B+, specifically related to netboot.
The foundation annouced upgrades for netboot support with this model, specfically the fact that netboot is activated by default on RPI 3B+ :


A per the announcement page on https://www.raspberrypi.org/blog/raspbe ... le-now-35/

1/ "So, without further ado, Raspberry Pi 3 Model B+ is now on sale for $35 (the same price as the existing Raspberry Pi 3 Model B), featuring:
A 1.4GHz 64-bit quad-core ARM Cortex-A53 CPU
Dual-band 802.11ac wireless LAN and Bluetooth 4.2
Faster Ethernet (Gigabit Ethernet over USB 2.0)
Power-over-Ethernet support (with separate PoE HAT)
Improved PXE network and USB mass-storage booting
Improved thermal management"

2/ "Raspberry Pi 3B was our first product to support PXE Ethernet boot. Testing it in the wild shook out a number of compatibility issues with particular switches and traffic environments. Gordon has rolled up fixes for all known issues into the BCM2837B0 boot ROM, and PXE boot is now enabled by default."


With this in mind we ordered the B+ and started testing...

Indeed, on RPi3B+ the PXE boot is enabled by default but the internal bootloader doesn’t enumerate USB devices where is the ethernet and fails to boot from the network when SD card is not present or doesn’t contain OS to boot. The only way to use PXE is to prepare SD card with updated RPi bootloader so it is not possible to use the SD card to boot and fall back to network boot if the SD card is not present or damaged.

Further:
There is nothing on the serial console when no SD card is present.
There is nothing on the serial console when we put a SD card that has just UART empty file to enable.

When we put SD card with UART and relatively old bootcode.bin (probably the same as is build in the RPi 3 B+ eprom) we got this:

Image

When we put newer bootcode.bin as is suggested in this GitHub issue by the guy who developed it (Gordon Hollingworth mentioned above in the product announcement I assume) , we got a normal PXE boot:

https://github.com/raspberrypi/firmware/issues/988

Image

Image
Image


The conclusion is this: the bootcode in the RPi3B+ units that we have is buggy. It was fixed later as we can see on Github, but this is not the version that we have in the units that we received.


Now the big question is: is there a way to fix this? Or just consider that netboot without an SD card is permanently broken on these boards?


:shock:

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12219
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: Netboot issues with mod 3 B+

Mon Jul 09, 2018 2:39 pm

I have notified all moderators.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2374
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Netboot issues with mod 3 B+

Mon Jul 09, 2018 2:55 pm

1. All 3B+s ship with the same SoC Boot ROM. There is no EEPROM. The only way to change the first stage loader is with a new spin of the chip, which would then be 2837C0.

2. A consequence of 1 is that if Gordon has fixed something so that booting works then that fix is available for all 3+s, not devices with a new ROM (because there isn't one).

3. A bug has been fixed in the firmware that was preventing a reboot from working. The fix hasn't reached the Raspbian firmware package yet but can be downloaded using "rpi-update".

4. The boot ROM does not access the UART - that feature is provided by bootcode.bin. If you are expecting to get some UART debugging output with no sdcard then the TFTP server must have a relatively up-to-date bootcode.bin that has been patched to enable the UART - see https://github.com/raspberrypi/linux/is ... -400649549.

If the above hasn't resolved your problem, please explain what you mean by the phrase "the internal bootloader doesn’t enumerate USB devices where is the ethernet", and post some logs from your dnsmasq/tftp server showing the (non-)boot sequence.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5963
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Netboot issues with mod 3 B+

Mon Jul 09, 2018 2:58 pm

PhilE wrote:
Mon Jul 09, 2018 2:55 pm
3. A bug has been fixed in the firmware that was preventing a reboot from working. The fix hasn't reached the Raspbian firmware package yet but can be downloaded using "rpi-update".
It should be in apt, I snuck it in last week.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1436
Joined: Sat Sep 10, 2011 11:43 am

Re: Netboot issues with mod 3 B+

Mon Jul 09, 2018 3:50 pm

Yes, as PhilE says above, there is a difference between 'bootcode' and 'bootrom' the boot sequence is:

1) BootROM executes on the chip, enumerates the USB devices and looks for a device matching the LAN9500 USB to Ethernet controller. It broadcasts a DHCP request and then gets a reply that includes the TFTP server address. Asks the TFTP server for bootcode.bin and then executes it.

2) Bootcode continues the ethernet connection from 1) and reads config.txt start.elf fixup.dat and then executes start.elf

3) start.elf continues the ethernet connection reading config.txt cmdline.txt, the dtb files, the linux kernel and filesystem if necessary. Then it kicks the ARM to start executing the linux kernel.


When you put 'just' the bootcode.bin onto an sdcard then it loads bootcode.bin from the sdcard, initialises the ethernet / DHCP / TFTP and continues from 2) above. So updates I make to bootcode.bin may fix problems at that initialisation stage but it cannot be 'programmed' into the Pi.

So if you cannot boot a Pi 3B+ without an SD card, it's nothing to do with the programming of the Pi 3B+ but more likely to be a problem with the network and server setup that the bootcode.bin is working around.

I would suggest that you run a tcpdump on the server while the Pi 3B+ is first plugged in (ignore the LEDs on the Pi 3B+, they are not enabled by the bootrom or bootcode)
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

palavrov
Posts: 33
Joined: Wed Jul 11, 2018 12:11 pm

Re: Netboot issues with mod 3 B+

Wed Jul 11, 2018 12:18 pm

Hi guys,

I'm Sephan's colleague working on our RPi based firmware.

During my research there was nothing into the dnsmasq/tftp logs even for bootcode.bin so my conclusion was that the bootrom doesn't load anything from tftp. Let'm double check that issue and will post here logs, wireshark traces, etc if it still occurs.

Atanas

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1436
Joined: Sat Sep 10, 2011 11:43 am

Re: Netboot issues with mod 3 B+

Wed Jul 11, 2018 3:09 pm

It's not clear what you mean here...

The bootcode.bin UART output Stephan has captured shows that it is correctly initialised and sending DHCP requests. I would suggest that your TCP dump is not working correctly.
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

palavrov
Posts: 33
Joined: Wed Jul 11, 2018 12:11 pm

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 3:24 am

Oh, yes - Stephan post wrong logs in the first message, this is what I captured on the UART:

There is nothing on the serial console when no SD card is present.
There is nothing on the serial console when we put a SD card that has just UART empty file to enable debug logs.

When we put SD card with UART and relatively old bootcode.bin (I think it was from 6th of May - that is the current RPi firmware package in buildroot) we got this:
Raspberry Pi Bootcode

Found SD card, config.txt = 0, start.elf = 0, recovery.elf = 0, timeout = 0
Trying USB
USB Host enumeration failed, no device detected at root

When we put newest bootcode.bin on the SD card, we got a normal PXE boot:
Raspberry Pi Bootcode
Found SD card, config.txt = 0, start.elf = 0, recovery.elf = 0, timeout = 0
Trying USB
Hub device found at addr 4, enumerating HUB
Initialise hub
Found 4 ports, multi_tt = 1
Setting interface 0
Enabling PORT POWER on port 1
Enabling PORT POWER on port 2
Enabling PORT POWER on port 3
Enabling PORT POWER on port 4
Waiting for devices to respond to reset
Found device on port 1
Found highspeed device
Hub device found at addr 5, enumerating HUB
Initialise hub
Found 3 ports, multi_tt = 1
Setting interface 0
Enabling PORT POWER on port 1
Enabling PORT POWER on port 2
Enabling PORT POWER on port 3
Waiting for devices to respond to reset
Found device on port 1
Found highspeed device
Device found: type = Ethernet adapter, addr = 6
Trying booting from Ethernet device addr 6
Initialise ethernet with MAC b8:27:eb:bb:fb:59
Wait for Link up
Failed to initialise link
Trying booting from Ethernet device addr 6
Initialise ethernet with MAC b8:27:eb:bb:fb:59
Wait for Link up
Failed to initialise link
Trying booting from Ethernet device addr 6
Initialise ethernet with MAC b8:27:eb:bb:fb:59
Wait for Link up
Link up
Sending DHCP request
Waiting for dhcp_reply
Done ARP for 200.201.168.192 got 7c:8b:ca:00:a4:2e
Read File: config.txt, 745 (bytes)




Raspberry Pi Bootcode
Read File: config.txt, 745
Read File: start.elf, 3913252 (bytes)
Read File: fixup.dat, 9735 (bytes)
[ 0.000000] Booting Linux on physical CPU 0x0

Because I didn't saw anything on the uart during pxe boot and there was no message in the tftp logs that at least bootcode.bin was served my conclusion was that bootrom doesn't work properly and we need to prepare SD cards with latest bootcode.bin in order to boot. Which means that we can't fall back to the pxe boot during factory installation or firmware upgrade i.e. it is a stop problem.

Based on your replies I want to double check and make sure (with wireshark, tcpdump, etc) that dnsmasq tftp receives and servers the expected file requests (or not). It can be that I didn't pay attention to something and conclude wrong based on the fact that exactly same setup boots perfectly with the previous RPi3 i.e. I was expecting smooth transition after switching to the new RPi3B+

Atanas

Janice357
Posts: 5
Joined: Wed Jun 27, 2018 11:46 am

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 6:06 am

This is pretty much a show-stopper for implementing PiServer with the new RPi 3B+.

Cold boot works fine. Everything is up-to-date in the chroot, firmware, etc.

Tried some slightly older bootcode.bin files from repo with no success they just broke PXE altogether.

Move to a more appropriate section if this is not a firmware issue.

palavrov
Posts: 33
Joined: Wed Jul 11, 2018 12:11 pm

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 7:42 am

gsh wrote:
Mon Jul 09, 2018 3:50 pm
1) BootROM executes on the chip, enumerates the USB devices and looks for a device matching the LAN9500 USB to Ethernet controller. It broadcasts a DHCP request and then gets a reply that includes the TFTP server address. Asks the TFTP server for bootcode.bin and then executes it.

I read somewhere during my research that problem could be because 10/100/1000 LAN7515 USB ID of the RPi3B+ doesn't match those of 10/100 LAN9544 which requires to load fixed bootloader.bin from sd card as workaround, could you please confirm that the USB ID is handled correctly in the bootrom?

I'll do a new test on our dev setup later today upgrading the bootcode.bin to the latest version on our tftp server - and really hope that this will solve the issue and all that is my miserable fault.

mfa298
Posts: 1387
Joined: Tue Apr 22, 2014 11:18 am

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 8:15 am

palavrov wrote:
Wed Jul 11, 2018 12:18 pm
During my research there was nothing into the dnsmasq/tftp logs even for bootcode.bin so my conclusion was that the bootrom doesn't load anything from tftp. Let'm double check that issue and will post here logs, wireshark traces, etc if it still occurs.
I bought a 3B+ shortly after they were released and netboot was one of the first things I tried, and yes I got it working with no sd card in sight.

First off it might be worth confirming it really is a 3B+ (metal cover over the WiFi and metal over the SoC).

The other potential issues I can think of which might stop you getting the initial dhcp and tftp requests are:

Some PoE splitters only pass two ethernet pairs through to the pi, with the Gigabit interface on the pi if it's plugged into a Gigabit switch that could stop the link coming up. Fixing the link speed might help although could lead to more issues.

Switch ports slow to allow packets; spanning tree, green Ethernet and various other things might mean the port doesn't pass packets for a few seconds, the bootrom will try doing it's dhcp and tftp requests much sooner than most other things try to talk to the network so it's possible the pi sends the requests but the switch is blocking them. Booting from the sd card initially might just put a slightly longer delay on before the requests are made (and may have better retry and backoff parameters)

palavrov
Posts: 33
Joined: Wed Jul 11, 2018 12:11 pm

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 8:56 am

mfa298 wrote:
Thu Jul 12, 2018 8:15 am
I bought a 3B+ shortly after they were released and netboot was one of the first things I tried, and yes I got it working with no sd card in sight.

Thanks!
The fact that it work for you gives me hope that will make it work for us too
mfa298 wrote:
Thu Jul 12, 2018 8:15 am
Some PoE splitters only pass two ethernet pairs through to the pi, with the Gigabit interface on the pi if it's plugged into a Gigabit switch that could stop the link coming up. Fixing the link speed might help although could lead to more issues.

Indeed, we are using PoE and this could be a problem ... but with bootcode.bin on SD card PXE boots as expected so probably the splitters are OK.
mfa298 wrote:
Thu Jul 12, 2018 8:15 am
Switch ports slow to allow packets; spanning tree, green Ethernet and various other things might mean the port doesn't pass packets for a few seconds, the bootrom will try doing it's dhcp and tftp requests much sooner than most other things try to talk to the network so it's possible the pi sends the requests but the switch is blocking them. Booting from the sd card initially might just put a slightly longer delay on before the requests are made (and may have better retry and backoff parameters)

That could be. With the old RPi3 we had too much troubles to get reliable pxe boot and drop the SD cards at all ... so the announce that RPi3B+ improved this (i.e. retrying instead of giving up) was giving us a hope to improve as well ... but instead we hit this problem that made me really (m/s)ad.

The boot process debug must be dramatically improved on the RPi to help developers catch whats wrong. Sometimes it can be something silly, sometime can be a bug - in all cases adding more visibility will improve the board. The fact that bootloader is closed source doesn't help too, I'm able to dig myself when hit such a problem but closed source just stops me - I don't want to steal Broadcom company secrets, just to do my job.

palavrov
Posts: 33
Joined: Wed Jul 11, 2018 12:11 pm

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 7:53 pm

Fixed!

RPi3B+ boots with dnsmasq running on my mac ... so probably I messed up something on the production setup and wrongly conclude that there is problem with the bootrom - my apologies guys.

jerrm
Posts: 194
Joined: Wed May 02, 2018 7:35 pm

Re: Netboot issues with mod 3 B+

Thu Jul 12, 2018 9:00 pm

palavrov wrote:
Thu Jul 12, 2018 7:53 pm
Fixed!

RPi3B+ boots with dnsmasq running on my mac ... so probably I messed up something on the production setup and wrongly conclude that there is problem with the bootrom - my apologies guys.
If no SD card is installed, fix the mmc rescan CPU utilization issue - see: viewtopic.php?f=28&t=211912

palavrov
Posts: 33
Joined: Wed Jul 11, 2018 12:11 pm

Re: Netboot issues with mod 3 B+

Fri Jul 13, 2018 8:17 am

jerrm wrote:
Thu Jul 12, 2018 9:00 pm
...
If no SD card is installed, fix the mmc rescan CPU utilization issue - see: viewtopic.php?f=28&t=211912

That's pretty weird ... fortunately in our setup we fallback to booting from SD cards to avoid endless netboot because we need to boot 100+ boards and sometimes it could take hours to boot all of them.

But because preparing hundreds SD cards with our buildroot based firmware is time consuming we are inserting blank SD cards and then when the board boots for first time the firmware downloads new sd card image from the tftp server and writes it to the sd card so on next boot it will quickly boot from there. Firmware upgrade works the same way - we just download new sd card image and rewrite the sd card.

Return to “Troubleshooting”