PoE HAT - USB Ports not working - over-current (FIXED - New Hardware Release)

[The_Raven]

We can talk now for a long time about it. But there are 2 things:
1. The PoE Hat ist faulty. Measuring shows this. And if you don't have USB-Devices, yes, it works, but what if you like to connect USB-Devices later? And what if you realize then that there is a problem? Yes, then your HAT is out of warranty.
2. It is not easy for any manufacturer to say "we made a fault". It is difficult for me to find the right words, english is not my native language. I find it great that the devs here say that it is faulty. This makes for me the difference between a good and a bad company. But i find it should told by a blog post.

[jamesh]

Yes, it doesn't work correctly in some circumstance. Finding those exact circumstances (we could not replicate the issue in house!) took a while with some great help from community members. Just as an anecdote, some community members sent guaranteed non-working devices, including the Pi3 they were attached too, to us for testing. We could not make them go wrong. Any of them, from multiple sources. That how odd this error is. Let me repeat that, kit that showed the error every single time they were turned on in the users home, when we tried them in the office, did NOT show the error. Exactly the same Pi3+ and PoE HAT, same SD card etc. Weird.

We do though, have a fix, which has been tested by the same community people with problems, and shown to work, and until that fix is applied, sales are suspended. It does take time to make the modification and refill the channels, hopefully just a few weeks. Sorry about the time it takes, but we need to get things right, and the process takes time.

[the_ewan]

Sure, I think everyone gets that. That's all totally fine.

The problem is that there isn't a blog post (and tweets etc.) simply saying what you've said here - there's a problem, sales are suspended, there will be a fix, it'll be a few weeks.

There's no disagreement about the actual issue, just whether the Foundation is being obvious enough about letting people know.

The only post on the blog about the PoE hat is "INTRODUCING THE POE HAT – AVAILABLE NOW!" and, well, it's just not.

A one line correction on that post would be useful. Right now it's easy enough to find the details if you already know there's a problem. But if you don't know there's a problem at all it's easy to miss it completely.

[sanhall]

I don't think there's been any problem with the communication between engineers and community in this thread, in fact it's been fantastic! I can appreciate how tricky this must be to work out, I have one compatible and one non-compatible Pi that were bought at the same time. For the record this issue also occurs with the pi camera, not just the physical USB ports, I'm sure you already guessed that though.

The lack of an official blog post is terrible, it just needs to state that the first batch had some compatibility issues and is being held back - we don't need a full replacement timeline immediately. "Out of Stock" is a lie, "Coming Soon" is a misleading half-truth. A 9+ page forum thread where the technical issues were debugged is not a source of truth for the current plan. Who knows which of the old posts during the investigation phase are still true, you'd expect there to be different decisions being made over time.

Without an official announcement to wrap it up the official forum staff are just being thrown under the bus - if the media picks up on it who else is there to quote? It's been a whole month since the hold on sales was mentioned in this thread, an official summary can't be seen as an unreasonable expectation.

[jamesh]

Sure, I think everyone gets that. That's all totally fine.

The problem is that there isn't a blog post (and tweets etc.) simply saying what you've said here - there's a problem, sales are suspended, there will be a fix, it'll be a few weeks.

There's no disagreement about the actual issue, just whether the Foundation is being obvious enough about letting people know.

The only post on the blog about the PoE hat is "INTRODUCING THE POE HAT – AVAILABLE NOW!" and, well, it's just not.

A one line correction on that post would be useful. Right now it's easy enough to find the details if you already know there's a problem. But if you don't know there's a problem at all it's easy to miss it completely.

I would expect any one doing a search will come to this thread just as easily whether the data were on the blog or not. Add a new blog post and it would fall off the front in a day, and never be seen again. A Google search for "raspberry Pi PoE HAT overcurrent" shows the first hit is this thread.

I'll see if the whether the original blog post can be updated to point to this thread though.

[The_Raven]

Interesting. But i think if it works or not is not so important, important is the measuring. It is not a clean DC output, it has a lot of ripples. Devs sould have a focus on this and trya to remove the ripples and make a clean DC output.
Only with a clean DC output runs the pi stable.

[jamesh]

Interesting. But i think if it works or not is not so important, important is the measuring. It is not a clean DC output, it has a lot of ripples. Devs sould have a focus on this and trya to remove the ripples and make a clean DC output.
Only with a clean DC output runs the pi stable.

We have fixed the issue.

[the_ewan]

A Google search for "raspberry Pi PoE HAT overcurrent"

Oh, come on - there's no way anyone can run that search until they not only know there's a problem, but something about the specific nature of the problem.

Someone who just wants a hat and has no idea why they're out of stock most places isn't going to run that search.

Google just "raspberry pi PoE hat" instead and the top hit is the Foundation's product page, and the second is the blog post which says 'available now', and neither of which mention the problem. This thread doesn't feature on the first page of results.

That's why those pages should have an update noting the issue.

[jamesh]

A Google search for "raspberry Pi PoE HAT overcurrent"

Oh, come on - there's no way anyone can run that search until they not only know there's a problem, but something about the specific nature of the problem.

Someone who just wants a hat and has no idea why they're out of stock most places isn't going to run that search.

Google just "raspberry pi PoE hat" instead and the top hit is the Foundation's product page, and the second is the blog post which says 'available now', and neither of which mention the problem. This thread doesn't feature on the first page of results.

That's why those pages should have an update noting the issue.

Replace "overcurrent" (which is the warning in the dmesg log) with "problem" and you still get a bunch of valid hits, with this thread being the third in the list. Replace with things like "not available" or "not working" get similar lists of hits. I don't think having a blog post would make the issue any more visible, but I have asked (the big boss), and it hasn't happened.

[Mortimer]

Surely it's up to the sales outlets how they present information about a product and what they say. If their online sales engine and website doesn't have any other appropriate item status than "out-of-stock", then what are they to do? It is technically correct.

[DjoeC]

I LIKE to be an early adopter but did not go for a pre-order or even an immediate order when available. Reason: Time between announcement and availability was too long - to me a warning sign. Then when they became available I waited a few weeks to first check for Raspberry POE hat Problems in a search engine. Luckily there were some postings therefore I haven't bought (yet).

The consequences are that my (outdoor mounting) project has been on hold for a long time.

The good thing is that support and openness in this thread is good and we are being informed. Thumbs up!
The bad thing is that version 1 (or is it 0.99) of the POE hat is still on sale without warning for the less wary people.

I hope to see a physically identifiable improved version coming soon! Good luck guys and galls!

[jamesh]

I LIKE to be an early adopter but did not go for a pre-order or even an immediate order when available. Reason: Time between announcement and availability was too long - to me a warning sign. Then when they became available I waited a few weeks to first check for Raspberry POE hat Problems in a search engine. Luckily there were some postings therefore I haven't bought (yet).

The consequences are that my (outdoor mounting) project has been on hold for a long time.

The good thing is that support and openness in this thread is good and we are being informed. Thumbs up!
The bad thing is that version 1 (or is it 0.99) of the POE hat is still on sale without warning for the less wary people.

I hope to see a physically identifiable improved version coming soon! Good luck guys and galls!

Time between announcement and launch is almost always just to ramp up production and fill up the channel. If you are familiar with the process of getting a production line (with associated test stations) up and running you will know how long this can take. This is one of the major reasons why almost every HW kickstarter ever has taken much longer to get to market that originally planned. The people doing them massively underestimate the timescales required, because on the whole they have no experience in it.

That said, we generally don't announce product in advance but in the PoE case, the announcement was forced by the appearance of the Pi3B+ and its PoE plug. Since we have limited staff, the PoE HAT ramp up had to happen after the Pi3B+ release (which of course was NOT announced in advance). Don't take the time taken as a warning sign, it's just the way it is. Ironically, had we spent more time on it, perhaps this mistake would have been detected prior to launch....

[Roken]

A Google search for "raspberry Pi PoE HAT overcurrent"

Oh, come on - there's no way anyone can run that search until they not only know there's a problem, but something about the specific nature of the problem.

Someone who just wants a hat and has no idea why they're out of stock most places isn't going to run that search.

Google just "raspberry pi PoE hat" instead and the top hit is the Foundation's product page, and the second is the blog post which says 'available now', and neither of which mention the problem. This thread doesn't feature on the first page of results.

That's why those pages should have an update noting the issue.

Then just try "pi poe hat problem", which I would generally do for a new product that I was planning on buying. It's my go to type search.

[the_ewan]

Or possibly the Foundation could just make it as easy as possible, rather than sticking to the 'bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying Beware of the Leopard' approach and expecting everyone else to work it out.

This obviously isn't a secret, there's clearly no cover-up, so what's the harm in making it a bit easier and simpler?

[martinrowan]

Yes, it doesn't work correctly in some circumstance. Finding those exact circumstances (we could not replicate the issue in house!) took a while with some great help from community members. Just as an anecdote, some community members sent guaranteed non-working devices, including the Pi3 they were attached too, to us for testing. We could not make them go wrong. Any of them, from multiple sources. That how odd this error is. Let me repeat that, kit that showed the error every single time they were turned on in the users home, when we tried them in the office, did NOT show the error. Exactly the same Pi3+ and PoE HAT, same SD card etc. Weird.

James, you or any team fathers team are welcome to pop round for a cuppa, inn order to test and measure any of the faulty pair of Pi+HAT, in my home network. Or you could send back a modified HAT on the combo that showed the issue to confirm that the proposed fix address all situations. Better to do that, than need to release a v3 version, if we managed to turn up something else. I know my home network has mains 240V cables running closer to the network cables than some other installations, but this shouldn't cause an issue. I assume the testing by the foundation team isn't just relying on lab PSU, or ideal network conditions and instead is experimenting with the real life variances that this type of hardware would be expected to handle.

Cheers Martin

[pantner]

Ok, so, I'm a little annoyed as well.

I was able to order this product on the 19th Sept, I had not thought to do any google searches to find out if there are any problems.

In between ordering (and shipping) of the HAT and it arriving i mentioned it on a forum thread i am a part of, a few people there made mention of power issues which then lead me here.

Since the only USB Device i use is a reciever for a wireless keyboard, i was hoping i would have good luck and it would work.

Unfortunately that is not the case. I was able to boot up the Pi (running the latest image of OSMC) and do the basic setup.
In the beginning it appeared to work, however when trying to play a specific video file, I had what I believe to be a kernel panic.
The same file then played ok after the Pi rebooted.

Went to bed, woke up in the morning, Pi was just frozen. I power cycled the Pi, within 30 seconds it had frozen again (while I was tying to browse my media library).
I switched back to a non-POE switch, plugged my USB power back in and it has been running flawlessly. (Apple iPad charger, not the best but it works)

The Pi, SD Card and HAT were all new out of the box (before this i had a Pi 2 running).

Is there anything that can be done?

[Burngate]

Is there anything that can be done?

Other than send it back for replacement, probably not.
It may depend on how much information we get about the gory details of what the problem turned out to be.

I'm intrigued by James's anectdote:

... some community members sent guaranteed non-working devices, including the Pi3 they were attached too, to us for testing. We could not make them go wrong.

Also, the report from someone managing to stop the problem by stalling the fan with his finger.

[jamesh]

Is there anything that can be done?

Other than send it back for replacement, probably not.
It may depend on how much information we get about the gory details of what the problem turned out to be.

I'm intrigued by James's anectdote:

... some community members sent guaranteed non-working devices, including the Pi3 they were attached too, to us for testing. We could not make them go wrong.

Also, the report from someone managing to stop the problem by stalling the fan with his finger.

Like I said, it's a weird one, but we do have a fix and are working on getting it out there asap.

[pantner]

Like I said, it's a weird one, but we do have a fix and are working on getting it out there asap.

Is there any point holding on to what I have now? Or should I just be returning it?

[jamesh]

Ok, so, I'm a little annoyed as well.

I was able to order this product on the 19th Sept, I had not thought to do any google searches to find out if there are any problems.

In between ordering (and shipping) of the HAT and it arriving i mentioned it on a forum thread i am a part of, a few people there made mention of power issues which then lead me here.

Since the only USB Device i use is a reciever for a wireless keyboard, i was hoping i would have good luck and it would work.

Unfortunately that is not the case. I was able to boot up the Pi (running the latest image of OSMC) and do the basic setup.
In the beginning it appeared to work, however when trying to play a specific video file, I had what I believe to be a kernel panic.
The same file then played ok after the Pi rebooted.

Went to bed, woke up in the morning, Pi was just frozen. I power cycled the Pi, within 30 seconds it had frozen again (while I was tying to browse my media library).
I switched back to a non-POE switch, plugged my USB power back in and it has been running flawlessly. (Apple iPad charger, not the best but it works)

The Pi, SD Card and HAT were all new out of the box (before this i had a Pi 2 running).

Is there anything that can be done?

Our recomendation if you are seeing issues is to return for a refund. The new version should be out soon.

[Paul Webster]

I saw one supplier announcing shipping (re-)starting yesterday.

Edit:
Correction ... the notification that Insaw was for a POE HAT starting to ship but it was not the one from RPT/RPF

[DjoeC]

James,

Like I said, it's a weird one, but we do have a fix and are working on getting it out there asap.

Could you provide some extra insight? Is it a problem with the Poe HAT, the 3B+, the firmware? And: how to recognize the fixed items with expected time to market? I would think many resellers will try to offload the problematic parts.....

Will there be any communication about specific cause and solution forthcoming?

Thanks!

[jamesh]

James,

Like I said, it's a weird one, but we do have a fix and are working on getting it out there asap.

Could you provide some extra insight? Is it a problem with the Poe HAT, the 3B+, the firmware? And: how to recognize the fixed items with expected time to market? I would think many resellers will try to offload the problematic parts.....

Will there be any communication about specific cause and solution forthcoming?

Thanks!

It's a hardware fault, requiring a fix to the HW. All board sent back to us by resellers will be fixed, so the majority of resellers will not be 'offloading problematic parts'. Future production runs will have a updated PCB anda version number bump. The fix is obvious*, so if anyone is selling on, then it will be easily spotted, although worth noting that for most people there is no issue.

Not sure of timescales, but must be pretty close.


* Although I just took a look at one, and had to have the fix pointed out. Once pointed out, its obvious!

[Burngate]

It's a hardware fault, ... All board sent back to us by resellers will be fixed ... Future production runs will have a updated PCB and a version number bump. The fix is obvious*, so if anyone is selling on, then it will be easily spotted.

* Although I just took a look at one, and had to have the fix pointed out. Once pointed out, its obvious!

So an "old" board can be fixed, but the "new" boards will have a different pcb, anyway?

I hope, when a blog comes along, we get a couple of photos to show us the fix. Possibly a "spot the difference" competition. With prizes, naturally. A cuddly toy for runners-up; what should the winner get?

[DougieLawson]

So an "old" board can be fixed, but the "new" boards will have a different pcb, anyway?

I hope, when a blog comes along, we get a couple of photos to show us the fix. Possibly a "spot the difference" competition. With prizes, naturally. A cuddly toy for runners-up; what should the winner get?

The obvious prize would be the re-designed "VNext" PoE HAT.