netview
Posts: 4
Joined: Tue Jul 02, 2019 10:15 am

Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 10:32 am

I have tested fw_0137a8 on my pi 4 and the temperature was about 3-4 C lower but my SSD (Samsung 860 Pro) was operating very slow :(
(the boot-Partition is on a micro-SD and root is fetched via SSD on USB 3.0)

with fw_013701:

root@raspbx:/usr/src# hdparm -t /dev/sda2
/dev/sda2:
Timing buffered disk reads: 1062 MB in 3.00 seconds = 353.91 MB/sec

with fw_0137a8:

root@raspbx:/usr/src# hdparm -t /dev/sda2
/dev/sda2:
Timing buffered disk reads: 1062 MB in 195.00 seconds = 1.81 MB/sec

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23083
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 11:09 am

netview wrote:
Tue Jul 02, 2019 10:32 am
I have tested fw_0137a8 on my pi 4 and the temperature was about 3-4 C lower but my SSD (Samsung 860 Pro) was operating very slow :(
(the boot-Partition is on a micro-SD and root is fetched via SSD on USB 3.0)

with fw_013701:

root@raspbx:/usr/src# hdparm -t /dev/sda2
/dev/sda2:
Timing buffered disk reads: 1062 MB in 3.00 seconds = 353.91 MB/sec

with fw_0137a8:

root@raspbx:/usr/src# hdparm -t /dev/sda2
/dev/sda2:
Timing buffered disk reads: 1062 MB in 195.00 seconds = 1.81 MB/sec
Ouch. We are looking in to it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2035
Joined: Thu Jul 11, 2013 2:37 pm

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 11:44 am

Please post a full sudo lsusb -v listing for your SSD. On 0137a8 with a Sabrent USB-SATA adapter, I get 350MB/s in the same test.
Rockets are loud.
https://astro-pi.org

tvjon
Posts: 708
Joined: Mon Jan 07, 2013 9:11 am

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 11:54 am

It seems to vary wildly with the drive!

Toshiba 2.5" integrated USB3:

/dev/sdb1: LABEL="tosh" UUID="72ACD728ACD3E61F" TYPE="ntfs" PTTYPE="atari" PARTUUID="9235e1ef-01"

$ sudo hdparm -t /dev/sdb1

/dev/sdb1:
Timing buffered disk reads: 314 MB in 3.01 seconds = 104.44 MB/sec


Seagate Barracuda 3.5" (much faster drive) via Sabrent USB3 docking station:

$ sudo hdparm -t /dev/sda1

/dev/sda1:
Timing buffered disk reads: 4 MB in 30.57 seconds = 133.98 kB/sec

Very strange.

$ sudo lsusb -v

attached.

HTH
Attachments
usb-v.txt.gz
(2.06 KiB) Downloaded 23 times

netview
Posts: 4
Joined: Tue Jul 02, 2019 10:15 am

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 12:28 pm

@jdb lsusb files attached
Attachments
lsubs.tar.gz
(4.63 KiB) Downloaded 16 times

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2035
Joined: Thu Jul 11, 2013 2:37 pm

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 12:38 pm

Are there any errors in dmesg after the drive's been accessed in a slow fashion?
Rockets are loud.
https://astro-pi.org

netview
Posts: 4
Joined: Tue Jul 02, 2019 10:15 am

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 12:41 pm

No - nothing!

tvjon
Posts: 708
Joined: Mon Jan 07, 2013 9:11 am

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 1:05 pm

jdb wrote:
Tue Jul 02, 2019 12:38 pm
Are there any errors in dmesg after the drive's been accessed in a slow fashion?
Attachments
dmesg-scsi.txt.gz
(3.91 KiB) Downloaded 26 times

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2035
Joined: Thu Jul 11, 2013 2:37 pm

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 1:32 pm

If you have a usb3.0 hub available, what happens when you plug the SSD with poor performance into a hub port instead of the Pi4 USB3.0 port?
Rockets are loud.
https://astro-pi.org

tvjon
Posts: 708
Joined: Mon Jan 07, 2013 9:11 am

Re: Raspberry Pi 4 temperature

Tue Jul 02, 2019 2:17 pm

$ sudo hdparm -t /dev/sda1

/dev/sda1:
Timing buffered disk reads: 2 MB in 30.24 seconds = 67.73 kB/sec
Attachments
hub-slow-test2.txt.gz
(15.75 KiB) Downloaded 10 times

Chimneyfactory
Posts: 23
Joined: Sun May 19, 2019 1:30 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 2:23 pm

I'm not sure if it's relevant, but in case it is : I've had some very strange things going on with USB3 & SSD / HDD on a Stock firmware Pi 4.

Prepping the disk :

Code: Select all

sudo gdisk /dev/sda
d
n
(take defaults)
w
then formatting :

Code: Select all

sudo mke2fs -t ext4 -L rootfs /dev/sda1
Causes the mouse icon to stop moving when mouse is moved (although mouse movement is still tracked).

This is repeatable, and happens if SSD / HDD connected through a powered USB 3 hub or direct to USB3 port. Tested with 4+ different SSD / HDD

Doesn't happen if the drive is connected to a USB2 port on the Pi4.

Doesn't happen if the gdisk step is replaced with using gparted instead.

Is weird.

trejan
Posts: 422
Joined: Tue Jul 02, 2019 2:28 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 2:32 pm

I'm seeing the same performance issues here but with an AirSpy SDR. No problems with the old firmware and downgrading fixes it. Nothing in dmesg or logs.

The reboot after firmware change doesn't seem reliable. It can hang and you need to pull the plug to restart it.

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2035
Joined: Thu Jul 11, 2013 2:37 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 2:32 pm

Chimneyfactory wrote:
Tue Jul 02, 2019 2:23 pm
I'm not sure if it's relevant, but in case it is : I've had some very strange things going on with USB3 & SSD / HDD on a Stock firmware Pi 4.

Prepping the disk :

Code: Select all

sudo gdisk /dev/sda
d
n
(take defaults)
w
then formatting :

Code: Select all

sudo mke2fs -t ext4 -L rootfs /dev/sda1
Causes the mouse icon to stop moving when mouse is moved (although mouse movement is still tracked).

This is repeatable, and happens if SSD / HDD connected through a powered USB 3 hub or direct to USB3 port. Tested with 4+ different SSD / HDD

Doesn't happen if the drive is connected to a USB2 port on the Pi4.

Doesn't happen if the gdisk step is replaced with using gparted instead.

Is weird.


The issue you are experiencing is unrelated to the new VLI firmware. Also, I am unable to reproduce it.
Rockets are loud.
https://astro-pi.org

Chimneyfactory
Posts: 23
Joined: Sun May 19, 2019 1:30 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 3:55 pm

jdb wrote:
Tue Jul 02, 2019 2:32 pm

The issue you are experiencing is unrelated to the new VLI firmware. Also, I am unable to reproduce it.
That's curious you can't reproduce it - apart from the Pi4 and the PSU (Official Pi one), I've swapped everything else out and it still happens. I've found a way around it (gparted) but its just weird.

Anyway, if it's unrelated to the firmware issue, I'll shut up now.

Thanks :)

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2035
Joined: Thu Jul 11, 2013 2:37 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 4:52 pm

Can you try this:

Code: Select all

sudo apt-get install pciutils
sudo setpci -s 01:00.0 0xD4.B=0x41
And see if that changes any of the SSD speeds?
Rockets are loud.
https://astro-pi.org

trejan
Posts: 422
Joined: Tue Jul 02, 2019 2:28 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 5:02 pm

jdb wrote:
Tue Jul 02, 2019 4:52 pm
Can you try this:

Code: Select all

sudo apt-get install pciutils
sudo setpci -s 01:00.0 0xD4.B=0x41
And see if that changes any of the SSD speeds?
This fixes my problem with the AirSpy SDR.

For anybody curious, this PCI config register change is disabling ASPM L1 on the VL805 so it only powers down one direction of the link now.

martinrowan
Posts: 64
Joined: Wed Aug 26, 2015 11:00 am
Location: Berkshire, UK
Contact: Website Twitter

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 5:52 pm

Seeing your post earlier today with the new firmware. I decided to give it a try to see if my power usage dropped. As I was at work at the time I connected over my VPN to run an SSH session. I followed the instructions to install the updated firmware and rebooted.

Code: Select all

history
  148  unzip vl805_update_0137a8.zip
  149  ls
  150  chmod a+x vl805
  151  sudo ./vl805 -w vl805_fw_0137a8.bin
  152  sudo reboot
The Pi didn't come back online. Upon returning home I found the console showing the successful end of the shutdown sequence:

Code: Select all

[ OK ] Reached target Shutdown.
[ OK ] Reached target Final Step.
[ OK ] Started Reboot.
[ OK ] Reached target Reboot.
	Stopping Restore / save the current clock...
See: Photo of the console

Disconnecting the power and reconnecting it, the Pi booted successfully. Flashing the old firmware and rebooting worked, re-flashing the new firmware and it hung at the reboot again.

Power usage with the new firmware: 3.3W from 3.6W without 4k enabled and 3.9W to 3.6W with 4k enabled. Sadly when in the Pi enclosure heat build-up seems pretty much the same, though may just take longer.
Last edited by martinrowan on Tue Jul 02, 2019 8:22 pm, edited 2 times in total.

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2035
Joined: Thu Jul 11, 2013 2:37 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 6:11 pm

trejan wrote:
Tue Jul 02, 2019 5:02 pm

This fixes my problem with the AirSpy SDR.

For anybody curious, this PCI config register change is disabling ASPM L1 on the VL805 so it only powers down one direction of the link now.
Ding Ding Ding! We have a winner. Yes, this is what the command does. There's some weird interaction between timers/latency entering L1 on the VLI chip's PCIe interface and the xHCI controller not being able to wake up the link. On USB2.0 ports, I've observed significant slowdowns when transferring data to an SSD but no hangs and on the USB3.0 ports we've been able to replicate the bus resets that other people are seeing.
Rockets are loud.
https://astro-pi.org

Raspberry Pi Rules!
Posts: 14
Joined: Thu Mar 06, 2014 8:32 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 7:00 pm

Someone on Reddit alerted me to the existence of this new firmware, but in addition to telling them my experiences, I've decided I may as well write them here in case they are at all helpful.

The temperature reduction is certainly noticeable, and I am very pleased with it! USB throughput I have not tested.

After flashing the firmware and rebooting, my Pi failed to turn on again. The lights were out. However, unplugging it from the power source and plugging it back in again fixed that. If that's a one-off issue, no big deal.

However, the next issue was more troublesome. I left my Pi idling in Discord with Chromium for about an hour an a half. The screen was locked with /usr/bin/dm-tool lock. When I returned, I found some graphical issue with Chromium - interface stuff missing. Resizing the window seemed to fix it.

But there was another issue, which was that the networking had stopped working. Chromium was returning ERR_NAME_RESOLUTION_FAILED, even for my router's page, and ping was returning nothing. I did sudo service networking reload to see if that would fix it, and it didn't.

At this point, the logical thing to do was to reboot the Pi. Turns out it didn't like that very much and it had a bit of a panic. I'm not going to go typing the whole thing out, but I took a photo, which you should be able to read in case it contains anything helpful.
panic.jpg
panic.jpg (232.54 KiB) Viewed 6600 times
I have now reverted to the older firmware and hope the issues were caused by this new firmware (as it could just be coincidence).

Aardappeltaart
Posts: 45
Joined: Wed Mar 02, 2016 11:32 am

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 7:54 pm

NEW firmware:
temperature dropped ~3°

nothing in syslog.

Had a bit fluctuating results in iozone with old AND new firmware on SD-card, USB HD and USB flash drive, running sequential test.

In new firmware lower (random) read/write on USB flashdrive (~15%)


BTW After reflashing old firmware, Pi did not reboot, I had to unplug and plug power-plug.

Chimneyfactory
Posts: 23
Joined: Sun May 19, 2019 1:30 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Tue Jul 02, 2019 8:35 pm

I tried the new firmware, and it went badly wrong for me.

Flashing the firmware to version 0137a8 returned the error "* Data mismatch". I retried, got the same error.

I rebooted, and saw errors on the screen at boot. Once booted, the USB keyboard and mouse weren't working.

SSH into the Pi, freshly downloaded the firmware, and flashed back to the original. No errors this time. Rebooted, mouse and keyboard now working!

Once bitten.. well, poke it with a stick and see what happens. I tried it again with my fresh download of the new firmware- perhaps the original was corrupted? I got the "* Data mismatch" error again. I tried to flash back to the original, still got the error. Reboot, and no keyboard and mouse once more.

SSH in again, flash with the new firmware, same copy as before - this time it works. Reboot, and I can use keyboard and mouse again!

Can also confirm that USB access speeds to an external HDD dropped from 87.26MB/sec to 33.27 kb/sec.

After the setpci command this increased very slightly to 134.52 kb/sec - still way slower than original.

Reverting back to the original firmware (got another "* Data mismatch" along the way and keyboard/mouse stopped again - reboot, reflash via ssh) and I've now got original speeds back.

trejan
Posts: 422
Joined: Tue Jul 02, 2019 2:28 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Wed Jul 03, 2019 1:08 am

Okay. I've done some more testing. I'm still seeing a problem with the new firmware even with the setpci tweak that turns off ASPM L1. Throughput is okay when ASPM L1 disabled but something in the new firmware is messing up timing or latency even with ASPM disabled.

I have this Airspy setup to be an ADS-B receiver which is linked to other receivers nearby for multilateration of aircraft that don't transmit coordinates. You need accurate timestamps of each sample for this to work and the new firmware does something with timing/latency that causes it to fail to synchronise with the other receivers.

I've tried the following:

New firmware with ASPM disabled (common clock on and off) <--- No throughput issues but multilateration doesn't work.
New firmware with ASPM L0s only <--- No throughput issues but multilateration doesn't work.
New firmware with ASPM L0s + L1 <--- Throughput issues and multilateration doesn't work.
Old firmware with ASPM disabled <--- No throughput issues issues and multilateration works.

itsmedoofer
Posts: 347
Joined: Wed Sep 25, 2013 8:43 am

Re: Raspberry Pi 4 - new VLI firmware issues

Wed Jul 03, 2019 7:24 am

Hi,

I had a quick play with the new firmware last night.

I saw a small reduction in throughput with a SSD in a USB 3 caddy, possibly 10Mb/Sec, 240 down to 230, I cant be too certain on this though.

Like the post above though I have seen odd behavior, I have a wireless keyboard mouse plugged into one of the USB 2 ports, if I plug my USB 3 HDD enclosure into a USB 3 port the mouse and keyboard start to become intermittent, I cant type anything reliably and the mouse movement is jerky.

Remove the enclosure the issue goes away, if I plug the enclosure into the USB 2 port I don't see the problem.

I didn't get time to try the SetPCi command.

I will see if I can do something a bit more scientific tonight..

phil995511
Posts: 21
Joined: Wed May 08, 2019 2:02 pm

Re: Raspberry Pi 4 - new VLI firmware issues

Wed Jul 03, 2019 4:08 pm

HI,

I have upgraded my Raspberry Pi 4 with the VL805 frimware version 000137a8

My system is instaled on a ssd connected by USB 3 (Samsung 850 Pro 128 Gb), it run without problem @176 Mo/s on average (from Gnome-Disk) and 338.45 MB/sec with the command

Code: Select all

hdparm -t /dev/sda
, as before with the old frimware (00013701).

My wireless mouse and keyboard from Logitech are conected by USB key and also work fine.

My system is on official Raspberry Pi 4 case, with passive heat sinks installed on the CPU, Wifi/Bluetooth chip, RAM, Motherboard Chipset & video controller. The difference of temperature is significant, currently @ 68/70° C versus 75° C with the old firmware on IDLE mode.

Thank you for this update functional at my home :)

I hope we will have soon a Raspbian OS in 64 Bits and Optimized frimware and/or OS to get a 4K display.

Thank you for everything and best regards to your team.
Last edited by phil995511 on Wed Jul 03, 2019 4:25 pm, edited 2 times in total.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2279
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 4 - new VLI firmware issues

Wed Jul 03, 2019 4:15 pm

Thank you for the feedback. We're going to hold back on a wider rollout until we understand (and have a fix for) the poor throughput others have seen, but I'm happy the update is working as intended for your use cases.

We are continuing to work on other performance and power improvements, but a 64-bit kernel for Pi 4 is likely to be released eventually.

Return to “Troubleshooting”