Dynamic profile VCore difference between RPi3 & CM3

[John Westlake]

I've spent the past week working on thermal issues (overheat) on the CM3 where we have not experienced issues with the RPi3.

In an effort to understand the cause of the CM3 thermal difference (about 10Deg higher) I've removed the WiFi BGA from the Rpi3 so that the hardware is the same from a software perspective.

We have noticed that VCore profile on the CM3 is “dynamically” 20mV to 50mV higher then the RPi3 running the same software (with the same attached hardware devices etc).

The Vcore on both the RPi3 and CM3 is 1.2V during low CPU overhead state (1.2000V appear the lowest dynamic VCore voltage), but when the CPU is operating under heavy load (on the same software task) the CM3 Vcore is 20mV to 50mV higher.

https://www.dropbox.com/s/gtqzfgthq0rwi ... 3.JPG?dl=0

In the graph linked to above you can see the difference (the multimeter samples in 1 Second intervals hence the “Quantise” timestep misalignment between graphs), but you can see that the CM3 VCore is always higher during higher CPU loads – this causing the higher SoC temperature.

Whats the reason for the VCore difference? – is it related to the PCB layout and PSU decoupling on the CM3 module? (verses the RPi3 PCB layout and decoupling) – or was extra voltage added to the CM3 VCore profile as a “design safety” margin to help with the stability of the customers designs (thus easing the customers PCB layout and decoupling requirements)?

Is there anyway with the CM3 to force the Kernel to set the same dynamic VCore profile as RPI3? I'd really like to reduce the internal heat dissipation – Temperatures with the RPI3 are OK, but with the CM3 heavy loads can trigger the Overheat warning – a temperature Delta of about 10Deg is just too much IMO.

To illuminate I/V voltage drops, these measurements where taken directly across C163 on the RPi and C42 on the CM3 (the VCore RT8088 main filter Cap).

[PhilE]

Manufacturing silicon chips is a bit like baking - you try hard to use the same ingredients and follow the recipe but your cakes rarely turn out to be identical. All CPUs show minor differences in doping levels that changes what can loosely be called the "speed" - some are "fast" parts and some are "slow" - and they have correspondingly different drive voltage requirements for a given target frequency. At boot time the firmware uses ring oscillators within the die to establish a safe operating voltage. Therefore you can't just compare a single Pi 3 with a single CM3 and expect that to tell you anything about the boards.

"vcgencmd get_config int | grep volt" shows you the result of the "AVS" tuning - both at idle and turbo - in microvolts above 1.2V, allowing you to compare different boards of the same type and build up a better picture of whether CM3 really does need more power.

[gsh]

Also,

Fast chips don't need as much voltage to run at the same speed and therefore use less power, but fast chips have greater static power consumption

Slow chips need more voltage but have less static power consumption...

The main difference between Pi3 and CM3 is the energy dissipation either through radiation (i.e. bigger board == more radiation), or through conduction (more connectors with larger quantity of metal == more conduction).

[John Westlake]

Thank you for the heads up on the onboard ring oscillator die calibration.

I've look at samples of the CM3 I have in the lab and created a splatter diagram of the over_voltage_avs / ring_osc(1) and ring_osc(2).

The ring_osc(2) results seems to have the most direct impact on over_voltage_avs / over_voltage_avs_boost.

However the splatter diagram highlighted some odd results (which suggest that “ring_osc(2)” only plays a part of the calibration routine) – I've highlighted the two “odd” CM3 module results:-

https://www.dropbox.com/s/jljgra7ns4ufx ... e.JPG?dl=0

Can you explain the reason for these two "odd" module results?

From the splatter diagrams I was able to select the “Best” and “Worst” CM3 modules and have been able to confirm the differences in VCore dynamic stepping levels (also our reference RPi3 VCore stepping):-

https://www.dropbox.com/s/5f0ayrtwae4dj ... 3.JPG?dl=0

Selecting the lowest and highest VCore “dynamic stepping” modules I performed thermal tests:-

Slowest CM3 Module (Highest Dynamic Voltage), Temp peaks at 49DegC:-

https://www.dropbox.com/s/v28lhf1iqwxsf ... 2.png?dl=0

Fastest CM3 Module (Lowest Dynamic Voltage step), Temp peaks at 48.5DegC:-

https://www.dropbox.com/s/btzycr3nbkixu ... 2.png?dl=0

Rpi3 Temp peaks at 44DegC:-

https://www.dropbox.com/s/ddxor8e47s5ta ... 3.png?dl=0

Now the question is why does the CM3 run hotter by about 5 DegC on the short-term peaks?

I'm now semi-retired, but one of my many consultant hats I was an analogue chip designer, so I spoke with my contacts in chip packaging and they confirmed what I suspected that the BGA packaging (especially internal wire bonded type devices) DO NOT by design conduct much heat via BGA pads (to prevent fatigue fractures caused by thermal cycling) – so there is very high thermal resistance between the Die / BGA / PCB interface making it unlikely that the smaller radiation area of the CM3 PCB can explain the short-term peak temperature difference between CM3 and Rpi3.

The CM3's smaller PCB area can explain the slightly higher longterm thermals but not on the 1-5 second scale... I would have thought that the thermal time constant of the BGA / PCB interface is just far to long to have any effect on the peak temperatures on the observed timescales.

1. Can you please confirm that the calibration variables and any coefficients used are the same between CM3 and RPI3 (I'm trying to understand the differences in the observed short-term peak temperatures)?

2. I noted that during different boots, the over_voltage_avs & over_voltage_avs_boost values are reported “randomly” in Hex or Decimal. One boot session they are displayed as Hex values the next boot session they could be displayed in decimal... could this is random variance indicate an issue?

3. 90% of the time, over_voltage_avs & over_voltage_avs_boost values are identical, is this expected?

4. Can you please give more details on the exact calibration variables and procedure used for over_voltage_avs & over_voltage_avs_boost values – how are these values calculated?

[John Westlake]

A gentle Ping please as we have overheat issues with CM3 modules during the update / booting process that does not appear to effect the RPi3.

[PhilE]

1. Yes, I believe so. Pi 3B+ gets special handling, but Pi 3 and CM3 should be treated identically.

2. Values above 65535 (0xffff) are displayed in hex, otherwise decimal is used.

3 & 4. I'm not ready to answer these questions yet.

[John Westlake]

Hi Phil,

Thank you for your reply, Doh - yes I should have realised the Decimal / Hex split - confirmed with a module (#9) right on the boarder (Now I feel stupid):-

Module #9:-

Boot 1

over_voltage_avs=62500 (62.50mV)
over_voltage_avs_boost=62500 (62.50mV)
ring_osc(1)=16.331MHz (@1.2000V)
ring_osc(2)=2.964MHz (@1.2000V)

Boot 2

over_voltage_avs=0x10c8e (68.75mV)
over_voltage_avs_boost=0x10c8e (68.75mV)
ring_osc(1)=16.329MHz (@1.2000V)
ring_osc(2)=2.965MHz (@1.2000V)

Boot 3

over_voltage_avs=0x10c8e (68.75mV)
over_voltage_avs_boost=0x10c8e (68.75mV)
ring_osc(1)=16.344MHz (@1.2000V)
ring_osc(2)=2.968MHz (@1.2000V)

Do you have any suggestion how to keep the thermals down during the software update process before the CPU thermal governor is loaded? The software update / installations takes around 3 minutes or so before the User space is loaded (I understand this is when the CPU thermal governor is loaded), during this period the CPU overheats and then its game over. I cannot get a clear answer from the software guys what they are doing (and why). Three minutes seems a crazy time to operate without the CPU governor enabled IMO - but this is how they worked with the RPi3...

Software guys like to blame hardware for any unexplained ill's and the standard answer is "Its OK with RPI3 so its must be a design problem with the CM3 implementation". I've done the test and I'm really at a lost here to explain the increased short-term thermals over the RPI3 - I'm hoping you can shed some light... the difference of 5-6 DegC pushes the unit over from a couple of very brief "half red" thermometer (80DegC+) indications on the RPi3 to Full Red (85Dec +) with the CM3 and then apparently we suspect the CM3 crashes... (well our software update / installation process fails).

[aBUGSworstnightmare]

Although I can't offer any particular help for solving your problem I would like to know if this could be solved by having a compute module based on BCM2837B0.
We can't change thermal performance of the existing CM3/3L by our baseboard design; only possibility would be adding heatsink to the chip which is not possible for some implementations.

[PhilE]

John,

Can you explain a bit about your "software update" mechanism? When running in mass storage device mode the CM doesn't run any of the normal thermal management code, but it also only runs at 250MHz with the ARMs off so there shouldn't be any significant heat generated. If instead this update is running under Linux then the thermal "governor" is already running - the VPU manages that behind the scenes; the CPU governor on the ARMs can request a switch to turbo mode, but the VPU will reduce the clocks in case of over-temp and under-volt.

[John Westlake]

Phil,

I'm afraid I cannot explain much as I only understand the hardware, but from the software developer:-

"Right after the system is initialized and kernel is loaded, we enter in the initramfs phase.

When updating, we stay in initram phase (which is a barebone OS system for early boot) for up to 3 minutes. My assumption is that the issue arises in this phase, since we don't have the userspace application that manages CPU speed (governor)

Once the update process has finished, we do switch-root and enter to a full fledged debian based system (userspace). This is when the cpu frequency scaling is set

Hope that's Comprehensive, if you need something else let me know"

I'm sure you will have more question, just let me know, or better still I'll try to get the software developer to communicate directly with you here.

[PhilE]

That's fairly comprehensive, but there a few points arising:
1. Can somebody confirm explicitly that you have no *_freq or *_turbo settings in config.txt?
2. Can you also confirm that the kernel is a stock rpi-4.14.y kernel built with bcm2709_defconfig? If not, what are the material difference? Has the default CPU governor been changed?

Even though there is no user-space cpufreq component, the governor is still in the kernel (unless you've disabled it - see 2 above). The default RPi kernel configurations choose "powersave" as the default governor, but Raspbian switches to "ondemand" fairly early provided the Shift key isn't held down. "powersave" keeps the clocks at the lower frequency (600MHz) which should limit heat generation.

The thermal governor runs on the VPU and is active before the ARMs execute their first instructions. The thermal limit is set to 85C, with a soft frequency reduction beginning 5C below that. You can reduce the upper limit using e.g. "temp_limit=70" in config.txt. That might be worth experimenting with.

[mikelangeloz]

Thanks, PhilE for your competent replies. To reply to your questions:

1. We do not have any *_freq or *_turbo setting in config.txt
We do actually have a turbo related config on cmdline, but its smsc95xx.turbo_mode=N which should not be related to the issue (and even unnecessary).

For sake of completeness here's the full cmdline:

Code: Select all

splash quiet plymouth.ignore-serial-consoles dwc_otg.fiq_enable=1 dwc_otg.fiq_fsm_enable=1 dwc_otg.fiq_fsm_mask=0xF dwc_otg.nak_holdoff=1 imgpart=/dev/mmcblk0p2 imgfile=/volumio_current.sqsh elevator=noop rootwait smsc95xx.turbo_mode=N bootdelay=5 logo.nologo vt.global_cursor_default=0 loglevel=0 consoleblank=0

And the whole config.txt

Code: Select all

initramfs volumio.initrd
gpu_mem=256
disable_splash=1
hdmi_force_hotplug=1

2. We use kernel 4.9.65, default config (except as USB quirks added, as per https://github.com/volumio/custom-kerne ... irks.patch)
Default governor has not been changed, although in userspace we use cpufrequtils with stock ondemand governor (with performance we were getting really high temps).

From my tests I noticed that temperatures tend to shoot up quite quick while booting, coming back to normal after reaching userspace (and idling). My concern is that our update procedure does a mere copy while in initramfs (so system load should be pretty low) but we do get really high temps in this scenario.

Looking forward for your thoughts

[PhilE]

Booting under systemd is an "aggressively parallel" process that can consume lots of power, but I am still surprised that it can generate so much heat so quickly. Please experiment with the temp_limit setting I mentioned in my previous post, if only to verify that the thermal limiting has an effect. If possible, could you also (but independently) try a null software update process that just waits for a reasonable amount of time, or perhaps just add a delay before the actual update, as a way of confirming that it is indeed the copy that is boosting the temperature and not something else happening at the same time?

[drphil3d]

I've been having a very similar issue, I have the CM3 in an aluminum enclosure with a heatsink that makes contact with the aluminum case, and it generates an incredible amount of heat.
I was hitting 86c and idling around 82c for the last few weeks

I've been doing testing with and without a heatsink in ambient air, under load it gets up to 80c without a heatsink, with a heatsink it doesn't pass 76c, however, generates a massive amount of heat.
I'm going to try limiting the temp in config.txt
turbo mode is disabled

[PhilE]

This the message I started:

Have either of you compared idle temperatures in your custom carrier boards with those in a stock CMIO board? The board on my desk is idling at about 45 in a room with an ambient temperature of 25C.

But then I ran my own thermal comparison with a Pi 3B.

This is the script spins:

Code: Select all

#!/bin/sh
while true; do true; done &
while true; do true; done &
while true; do true; done &
while true; do true; done &
while true; do vcgencmd measure_temp; sleep 1; done

On the 3B, starting at around 44C it hit 60C in 2 minutes, and by 6 minutes it had reached 66.6C (it's a beast of a test). Switching to the CM3 + CMIO (v1.2) board I repeated the test. It started the run at 42C, stormed past 66.6C in 20 seconds and triggered the overtemp warning at 80C in 55 seconds. This is with no peripherals attached except an HDMI monitor and a serial port. A second run with no monitor started at 38C and took 68 seconds to reach 80C.

With only two spinning threads it reach 66.6C in 2 minutes, then very slowly continued to climb - I stopped the test at 6 minutes and a temperature of 68.8C. For the record, my CM3 is a fast part - over_voltage_avs is 62500.

The CM3 PCB is less than half the size of the Pi3 and about 60% of the thickness, so (without any training in the subject) I would expect thermal dissipation to be slower, but is that difference enough to account for the large differences observed in testing?

[PhilE]

The verdict at Raspberry Pi is that, without a heatsink, this rapid temperature rise is unavoidable under load. Contrary to the assertions of the original poster, the ball grid array is actually very good at conducting heat, and the higher thermal mass of the Pi3B and Pi3B+ due to larger, thicker PCBs and numerous metal connectors means the temperature spreads more evenly and rises more slowly. The thermal limiter will stop the temperature from running away, but if a lower operating temperature is important then the ARM core clock rate can be reduced with the "arm_freq=..." config.txt setting.

[John Westlake]

I've been having a very similar issue, I have the CM3 in an aluminum enclosure with a heatsink that makes contact with the aluminum case, and it generates an incredible amount of heat.
I was hitting 86c and idling around 82c for the last few weeks

I've been doing testing with and without a heatsink in ambient air, under load it gets up to 80c without a heatsink, with a heatsink it doesn't pass 76c, however, generates a massive amount of heat.
I'm going to try limiting the temp in config.txt
turbo mode is disabled

Yes I agree, I strong feel there is something more fundamentally wrong here, we have added a small 14mmx14mm heatsink to the CM3 CPU and we STILL have thermal issues where as the RPi3 with no heatsink is perfectly fine!!!

I've been a designer for over 30 years and worked on many IC and hardware designs - my "Gut" engineering feeling that I've relied on all these years says there is something more we are missing, something is not working as expected here...

[John Westlake]

This the message I started:

Have either of you compared idle temperatures in your custom carrier boards with those in a stock CMIO board? The board on my desk is idling at about 45 in a room with an ambient temperature of 25C.

But then I ran my own thermal comparison with a Pi 3B.

This is the script spins:

Code: Select all

#!/bin/sh
while true; do true; done &
while true; do true; done &
while true; do true; done &
while true; do true; done &
while true; do vcgencmd measure_temp; sleep 1; done

On the 3B, starting at around 44C it hit 60C in 2 minutes, and by 6 minutes it had reached 66.6C (it's a beast of a test). Switching to the CM3 + CMIO (v1.2) board I repeated the test. It started the run at 42C, stormed past 66.6C in 20 seconds and triggered the overtemp warning at 80C in 55 seconds. This is with no peripherals attached except an HDMI monitor and a serial port. A second run with no monitor started at 38C and took 68 seconds to reach 80C.

With only two spinning threads it reach 66.6C in 2 minutes, then very slowly continued to climb - I stopped the test at 6 minutes and a temperature of 68.8C. For the record, my CM3 is a fast part - over_voltage_avs is 62500.

The CM3 PCB is less than half the size of the Pi3 and about 60% of the thickness, so (without any training in the subject) I would expect thermal dissipation to be slower, but is that difference enough to account for the large differences observed in testing?

I've added a heatsink to the CM3 and STILL experience overheat in a very shorttime where as we have no heat issue with RPI3 without heatsinking - my engineering experience is saying that something unexpected is happening here.

I'd like to replace the BCM2837 on a CM3 module with one pre-programmed with the RPI3 device ID to see if there is something odd happening at the Kernel level - to save me having to remove and reball a BCM2837 from a RPi3, is there anychance of sending me a couple of pre-programmed "RPI3" BCM2837 - or are they "programmed" in situ on the production line during final PCBA testing with Device ID?

My understanding is that the BGA package of the BCM2837 (RPI3 not RPI3+) is internally wirebonded? if so then these have high thermal resistance - not all BGA's are the same.

Do you have the thermal Resistance figures for the BCM2837 BGA package used on the RPi3 / CM3? (not the BCM2837B0 used on the RPI3+).

[John Westlake]

A gentle Ping to my earlier post above please ...

I've noticed that there are more reports of CM3 Overheat issues:-

viewtopic.php?f=98&t=175523

[PhilE]

Yes CM3 can heat up quickly as discussed at length, but so far there is no evidence that the thermal governor doesn't work. I have made a number of suggestions - reducing the ARM frequency, lowering the thermal limit - neither of which you have responded to. Perhaps now would be a good time to do so.

The other thread you've jumped on is being investigated as an undervoltage issue rather than overheating (although heat may be a contributing factor), but that may also be true in your case, so you could try adding some additional voltage:

Code: Select all

over_voltage=2  # +50mV

You can also post the output of "vcgencmd otp_dump" to help us spot any unusual OTP values.

[John Westlake]

Phil,

I'm away from the Lab until later this afternoon, as soon as I return I'll run the tests.

Maybe the title of this thread is causing some confusion - but the reason behind the thread is to understand why we have serious overheat issues with CM3 where we have non with RPi3.

In the overheat thread I posted:-

"It has been suggested that the thermal performance difference between RPi3 and CM3 is due to the smaller PCB heatsinking area of the CM3 module - but this FAILS to explain why we still have thermal issues when a heatsink is added to the CM3 - I cannot believe that a BGA device mounted on standard FR4 PCB laminate (RPI3) has better thermal performance then a BGA with added heatsink but mounted on a smaller PCB area"

I believe that the addition of a heatsink attached directly ontop of the BGA package must rule out PCB thermal dissipation difference as the reason for overheat we are experiencing between RPi3 and CM3...

[mikelangeloz]

Hi Phil,
if the results of those tests might be beneficial, there you have them:

1 - Thermal limit to 70
When booting with temp_limit=70 in config.txt, the thermal warning icon is constantly there (meaning we never go below 70), boot operation is slightly slower (as you would expect since the CPU is throttled down) but it completes correctly. There is no sign of instability due to throttling. I am preparing a test update to verify if being in such situation results in issues during update (but that's a side effect, not really related to the topic).

2 - Limiting frequency with arm_freq
When booting with arm_freq=800, I still see the thermal warning icon quite often, sometimes being full (85) but most of the time is on the "half position" (80). I've tested extensively if reducing frequencies is helping, but in fact it's not.

3 - Over voltage
Adding over_voltage=2 to config.txt, results in a slight increase in temperature. Without idling temp is at 77.9, with it's 79.5. Measured with
/opt/vc/bin/vcgencmd measure_temp

4 - OTP Dump

There you have it:

Code: Select all

08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1220000a
18:1220000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:3b9797b0
29:c468684f
30:00a020a0
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000000
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

If there is any other info that might be useful, let me know

[PhilE]

Can you please state for 2 and 3 whether the boot-time updates complete successfully or not?

[mikelangeloz]

Hi,
let me clarify: I did the assumption that updates might be impacted by CPU throttling in early initramfs stage (if temp was high). This is just a side effect that I am investigating now, while the topic of discussion is about temperatures on CM3.

[John Westlake]

Although I can't offer any particular help for solving your problem I would like to know if this could be solved by having a compute module based on BCM2837B0.

Does anyone know if the BGA footprint is the same for the BCM2837 & BCM2837B0 - if they are the same then I can transplant a BCM2837B0 onto a CM3 module and run some thermal tests...

Our XRAY machine has damaged to its shielding so I'm not to keen on using it until the replacement shield arrives...