LdB
Posts: 1172
Joined: Wed Dec 07, 2016 2:29 pm

Re: CNTP interrupt not firing

Thu Feb 28, 2019 4:26 pm

bzt wrote:
Wed Feb 27, 2019 10:49 am
Do you route the timer IRQ to one core only, or to all at once? I understand that the scheduler gets called individually on all cores at the end, I'm just curious what's happening in between.
The easiest is to route the QA7 one to each core and each core handles it's irq, they will all have to share the same frequency because the prescaler and divider is global from the 38.4Mhz.

Reference section: "Symmetric OSes: master/slave and the alternatives" in
https://www.embedded.com/design/mcus-pr ... ext-Part-2

What it is essentially setting up is a "load balancing SMP RTOS" with the desired outcome
A superior solution to the two implementations presented above is an SMP RTOS, that in addition to “load balancing” different tasks, also load balances kernel services. By threading the kernel services and allowing them to run in parallel on different microprocessors, the possibility of an unavailable kernel service decreases dramatically.

User avatar
Ultibo
Posts: 158
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: CNTP interrupt not firing

Thu Feb 28, 2019 10:56 pm

bzt wrote:
Wed Feb 27, 2019 10:49 am
Every higher priority level have more timeslices than the one beneath, and the top 3 levels are uninterruptible, meaning when the IRQ handler is executed, it won't call the scheduler
Hi bzt,
What is your rationale for choosing to make the top 3 priority levels uninterruptible?

Why not (as we do in Ultibo) simply have a flag or a system call (depending on how sophisticated the environment is) that controls preemption so any task can disable and reenable preemption on demand without the impact of also disabling interrupts. So when the timer IRQ is fired it simply checks the flag (one per CPU) to see if it should invoke the scheduler.

While you could argue that such a scheme allows any task to hog the CPU we aren't talking about a general purpose OS here and in a bare metal environment there is greater scope to allow the designer such flexibility. In reality even Linux and Windows allow any task to increase their priority (the nice value or thread priority) to a level that effectively makes the system unusable.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: CNTP interrupt not firing

Sun Mar 03, 2019 1:32 pm

Hi,
LdB wrote:
Thu Feb 28, 2019 4:26 pm
The easiest is to route the QA7 one to each core and each core handles it's irq, they will all have to share the same frequency because the prescaler and divider is global from the 38.4Mhz.
That's exactly what I do. This way I have per core counters and per core scheduler IRQ. Having the same freq is actually helping, makes task sync easier (I also set the crystal as clock source instead of APB) :-)

Ultibo wrote:
Thu Feb 28, 2019 10:56 pm
What is your rationale for choosing to make the top 3 priority levels uninterruptible?

Why not (as we do in Ultibo) simply have a flag
Actually that acts like a flag, technically it works exactly as you described. In the ISR where you use "if (rtflag)" I use "if (priority < 3)" to decide if the task is uninterruptible. I already had the priority queues, and a nice ACL system to check if a user application can be put in one particular queue with the setprio() system call, and I didn't wanted to introduce a new system call. I expect varying tasks (not all known at start), in that regard this is a bit like a general purpose OS, so I also had to make sure that a lower priority task won't make a higher priority task starving (hog the CPU as you put it very expressively). Pretty much that's all the reasons behind.

I have 3 top levels because my priority queues are like this: 0-system (only for emergency tasks, and for the built-in debugger if compiled in), 1-real time tasks, 2-device driver tasks, 3-servers, (3+)-applications. I have other reasons than scheduler to have level 2 and 3 in a micro-kernel architecture (drivers are special tasks, and servers are user space parts of the kernel).

Cheers,
bzt

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: CNTP interrupt not firing

Tue Mar 05, 2019 11:12 am

Circle is working with QEMU 3.1.0 now with AARCH = 64 (including CNTPNSIRQ). I have replaced the BCM283[567] System Timer implementation with the ARMv7 Generic Timer (RPi Core Timer) for that purpose. The source is on the branch "core-timer" at the moment.

That means the Core Timer is already working in QEMU 3.1.0 with "-M raspi3" at least for core 0. The counter frequency is different (62.5 MHz instead of 19.2 MHz), but this can be handled using the CNTFRQ register. Multi-core support and SD card is working too. The RPi USB host controller interface emulation is not available in this QEMU version.

LdB
Posts: 1172
Joined: Wed Dec 07, 2016 2:29 pm

Re: CNTP interrupt not firing

Tue Mar 05, 2019 3:38 pm

It is actually 38.4MHz from the 19.2Mhz clock on real hardware

https://www.raspberrypi.org/documentati ... rev3.4.pdf
Section 4.11
The code has a single local timer which can generate interrupts. The local timer ALWAYS gets its timing pulses from the Crystal clock. You get a 'timing pulse' every clock EDGE. Thus a 19.2 MHz crystal gives 38.4 M pulses/second.

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: CNTP interrupt not firing

Tue Mar 05, 2019 5:04 pm

LdB wrote:
Tue Mar 05, 2019 3:38 pm
It is actually 38.4MHz from the 19.2Mhz clock on real hardware

https://www.raspberrypi.org/documentati ... rev3.4.pdf
Section 4.11
The code has a single local timer which can generate interrupts. The local timer ALWAYS gets its timing pulses from the Crystal clock. You get a 'timing pulse' every clock EDGE. Thus a 19.2 MHz crystal gives 38.4 M pulses/second.
I used the 64-bit per Core Timer (described in sections 3.1 and 4.3) and in the ARMv8 ARM, not the Local Timer. It generates 19200000 clock ticks per second with a prescaler of 0x80000000. The CNTFRQ_EL0 register is also set to 19200000.

LdB
Posts: 1172
Joined: Wed Dec 07, 2016 2:29 pm

Re: CNTP interrupt not firing

Wed Mar 06, 2019 1:08 am

Confused with your answer rst, I read it and have always coded the source for the clock in 3.1 and 4.3 can only be the 19.2mhz xtal (which will actually be 38.4Mhz as per above) or the bus clock as per 3.1.1

I know you have running code so that makes me think there is another clock I haven't found so could you outline your register setup to run from 19.2Mhz because I have definitely only found 38.4Mhz so far.

User avatar
Ultibo
Posts: 158
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: CNTP interrupt not firing

Wed Mar 06, 2019 6:35 am

LdB wrote:
Wed Mar 06, 2019 1:08 am
I know you have running code so that makes me think there is another clock I haven't found so could you outline your register setup to run from 19.2Mhz because I have definitely only found 38.4Mhz so far.
QA7_rev3.4.pdf page 9
Section 4.2 Control register

Bit 8: Core timer clock source
If clear the 64-bit core timer pre-scaler is running of the Crystal clock

Bit 9: Timer increment
If clear the 64-bit core timer is incremented by 1
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: CNTP interrupt not firing

Wed Mar 06, 2019 7:17 am

LdB wrote:
Wed Mar 06, 2019 1:08 am
I know you have running code so that makes me think there is another clock I haven't found so could you outline your register setup to run from 19.2Mhz because I have definitely only found 38.4Mhz so far.
First the initialisation code:

Code: Select all

#define HZ	100

u64 nCNTFRQ;
asm volatile ("mrs %0, CNTFRQ_EL0" : "=r" (nCNTFRQ));
assert (nCNTFRQ % HZ == 0);
m_nClockTicksPerHZTick = nCNTFRQ / HZ;

u64 nCNTPCT;
asm volatile ("mrs %0, CNTPCT_EL0" : "=r" (nCNTPCT));
asm volatile ("msr CNTP_CVAL_EL0, %0" :: "r" (nCNTPCT + m_nClockTicksPerHZTick));

asm volatile ("msr CNTP_CTL_EL0, %0" :: "r" (1));	// enable timer
Reloading CNTP_CVAL_EL0 in the IRQ handler:

Code: Select all

u64 nCNTP_CVAL;
asm volatile ("mrs %0, CNTP_CVAL_EL0" : "=r" (nCNTP_CVAL));
asm volatile ("msr CNTP_CVAL_EL0, %0" :: "r" (nCNTP_CVAL + m_nClockTicksPerHZTick));
This definitely leads to 100 IRQs/sec on the RPi 3B+. I did not change the registers, which act on the clock source.

LdB
Posts: 1172
Joined: Wed Dec 07, 2016 2:29 pm

Re: CNTP interrupt not firing

Wed Mar 06, 2019 8:22 am

Well look at that it is another clock in the ARM8 core itself using only ARM8 registers, I thought all that had been ripped out with the GIC.

So it looks like it is sources from the same clock but has it's own divisor (per core) and interrupt??

What ARM8 registers do you bash to trap the interrupt?
http://infocenter.arm.com/help/index.js ... GDFGH.html
Does not mention how to trap the interrupt.

That would actually allow bzt to setup exactly what he wanted with hardware :-)

Something making me scratch my head, if that is designed for the scheduler why is it EL0?
Oh unless you pass the trap up to EL1 on a register or is that a backdoor to the switcher frequency to allow timekeeping?

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: CNTP interrupt not firing

Wed Mar 06, 2019 9:04 am

LdB wrote:
Wed Mar 06, 2019 8:22 am
What ARM8 registers do you bash to trap the interrupt?
I only set bit (1 << 1) in the register at 0x40000040. Of course IRQs must be enabled in PSTATE.I and VBAR_EL1 must be programmed with a valid exception vector structure. I have set the entries at offset 0x80 and 0x280 as IRQ stubs. My code runs at EL1.
Something making me scratch my head, if that is designed for the scheduler why is it EL0?
The access to these *_EL0 registers may be trapped using CNTKCTL_EL1. I suppose, in a real system, which implements protection mechanisms, you do not have access to them at EL0.

LdB
Posts: 1172
Joined: Wed Dec 07, 2016 2:29 pm

Re: CNTP interrupt not firing

Wed Mar 06, 2019 9:32 am

rst wrote:
Wed Mar 06, 2019 9:04 am
I only set bit (1 << 1) in the register at 0x40000040. Of course IRQs must be enabled in PSTATE.I and VBAR_EL1 must be programmed with a valid exception vector structure. I have set the entries at offset 0x80 and 0x280 as IRQ stubs. My code runs at EL1.
That is nCNTPSIRQ_IRQ so that now makes perfect sense.
Your vectors are Current EL with SP0 and Current EL with SPx only one would be needed depending if you set spsr_el2 with 0x3c5 or 0x3c4

I tested the hypervisor one and that timer works as well ... so lots more clocks :-)
rst wrote:
Wed Mar 06, 2019 9:04 am
The access to these *_EL0 registers may be trapped using CNTKCTL_EL1. I suppose, in a real system, which implements protection mechanisms, you do not have access to them at EL0.
Yeah so looking at the linux aarch64 timekeeping code the interrupt is trapped at EL1 but EL0 apps can access the register, so it is sort of a backdoor rather than the usually virtual structure for time tick in memory update.

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: CNTP interrupt not firing

Wed Mar 06, 2019 9:52 am

LdB wrote:
Wed Mar 06, 2019 9:32 am
Your vectors are Current EL with SP0 and Current EL with SPx only one would be needed depending if you set spsr_el2 with 0x3c5 or 0x3c4
Yes, the vector at 0x80 would probably be enough. My main code runs in EL1t using SP_EL0. The IRQ handler in EL1h using SP_EL1.
Yeah so looking at the linux aarch64 timekeeping code the interrupt is trapped at EL1 but EL0 apps can access the register, so it is sort of a backdoor rather than the usually virtual structure for time tick in memory update.
Really?

LdB
Posts: 1172
Joined: Wed Dec 07, 2016 2:29 pm

Re: CNTP interrupt not firing

Wed Mar 06, 2019 10:15 am

Yep if I am reading it right there is a switch you can virtualize them to memory in arm_arch_timer.c with

Code: Select all

static bool arch_timer_use_virtual = true;
the Pi3 will be set as

Code: Select all

static bool arch_timer_use_virtual = false;
As it uses real registers so the ARM8 seems to have been designed a bit with linux in mind.

Return to “Bare metal, Assembly language”