LdB
Posts: 1209
Joined: Wed Dec 07, 2016 2:29 pm

Re: Interrupts time and memory?

Thu Jun 22, 2017 7:54 pm

It was actually the set_LED routine that was crashing and there is a whole story to it.

The first problem was we butchered the HYP_MODE drop down routine somehow it has a glaring mistake in it between the current one and the old original one. Not sure if I did it at some point or I copied from you or what but it differs from the original I gave you. It played into a problem with the optimizer because you used commas when setting the array in set_LED .... Do you know what the comma does on a line of C/C++ code?

There are still a lot of errors in your code and some look to be quite fatal. I would be very slow about trying to bring the string routines up they are very bugged.

Interrupt working files: http://dropcanvas.com/#6of1hr0Lh0Kj7s

Anyhow I need to check I have the interrupt is working properly so I would like you to carefully do the following steps

1.) Try the kernel8-32.img in the linked zip file on your SD card hopefully we have a flashing light and a blank screen AKA interrupts are working
2.) I would like you to edit pi3.bat to have your path to your g++ compiler leaving all the rest alone and run the compile and make a new kernel8-32.img. Test the file and check it works. If it works move to step 3.
3.) Remove the kernel32-8.img off the SD card it will take precedence over your kernel.img file because of this

RPi3 Boot sequence:
Check for kernel8.img and if found, load it and boot in 64bit mode.
If not found, check for kernel8-32.img and if found, load it and boot in 32bit mode
If not found, check for kernel7.img and if found, load it and boot in 32bit mode
If not found, check for kernel.img and if found, load it and boot in 32bit mode

3.) Move the files (boot.s, mailbox.c, mailbox.h, timePiOS.c, timePiOS.h) and main.c to your repo in the right places. Alter your make to use main.c rather than kernel_main.c and compile. Compile and see if the repo now produces working code in kernel.img.
4.) If it works change back to your kernel_main and compile and fix everything :-)

Good luck.

Makogan
Posts: 71
Joined: Tue May 16, 2017 9:17 pm

Re: Interrupts time and memory?

Fri Jun 23, 2017 12:13 am

O.O it was that one line...
(The bx lr line in your Hyp to SVC subroutine that I added like an idiot... OMG!!!!!!!!!!!!)

THANK YOU!

I need to ask 2 questions though (Sorry, I told you I'd kept spamming)

First, how is that subroutine returning? Like if I look at your subroutine, the last statement is mov r0, #1

How does it return to it's calling code without an explicit branch instruction?

Second, why is it that being in Hyp mode prevents you from going into lower priviledged modes? that makes no sense in my head...

On any case thank you for helping me, I know it was my fault and that I wasted your time, I am sorry.

LdB
Posts: 1209
Joined: Wed Dec 07, 2016 2:29 pm

Re: Interrupts time and memory?

Fri Jun 23, 2017 5:18 am

Okay first up most of what I started with was from the ARM site they have a reasonable baremetal guide
http://infocenter.arm.com/help/topic/co ... essors.pdf
The Pi3 being an ARM8 CortexA-53 it is the most applicable reference. Then I sponged from Ultibo, Peter Lemon and David :-)
Makogan wrote:First, how is that subroutine returning? Like if I look at your subroutine, the last statement is mov r0, #1
Okay this is about security what they are trying to do is setup what on an Intel processor they call rings of security. So the O/S system runs at the highest privilege, Device drivers then App run at a lower and finally Users run at the lowest. So to switch security levels they want to be able to monitor it which is why you can't just flick a register.

We now need to jump to this question
Makogan wrote: Second, why is it that being in Hyp mode prevents you from going into lower priviledged modes? that makes no sense in my head...
This naming in ARM security I also find weird having come from Intel. It's actually covered reasonably well in wikipedia under protection ring
https://en.wikipedia.org/wiki/Protection_ring
They talk about the ARM7 and here is the quote
The ARM v7 architecture implements three privilege levels: application, operating system, and hypervisor. Unusually, level 0 (PL0) is the least-privileged level, while level 2 (PL2) is the most-privileged (hypervisor) level
So that is why the Pi starts in PL2 (HYP_MODE) because it should start in highest security. Originally in history it didn't we got the ARM in SVC_MODE but they changed the bootloader.

So the bit that first got me is security terms on the ARM are backwards when you are in secured mode operation what you are saying is you can't access some things (AKA real world you have less privilege). So the terms ARM uses are backwards to how we think and Intel .. why would you do that ARM :-).

So when going out of HYP_MODE it is a serious thing you are about to give up all your privileges and they don't want some random O/S crash giving up the privilege. So the special way to ask for that switch is with special opcodes
"msr ELR_hyp,r0
eret"
Makogan wrote: How does it return to it's calling code without an explicit branch instruction?
eret is a special return
http://infocenter.arm.com/help/index.js ... JEHII.html
When executed in Hyp mode, ERET loads the PC from ELR_hyp and loads the CPSR from SPSR_hyp
You obviously can't load ELR_hyp is you aren't in HYP_MODE and now go look at the code again :-)

So now you may ask, why get out of HYP_MODE it's the most privilege. Well the answer is in what you were doing trying to setup the interrupts. You can't setup the registers for some of the secured modes from within HYP_MODE .. why you can't is a question to ask ARM (I live in hope you can but I have not been smart enough to work it out). So HYP_MODE becomes a pain in the butt because you need that special sequence to get out of it just to go and change a few registers in the IRQ_MODE and FIQ_MODE. Then you have to make sure you setup the service to get back into it. There are only some security thing you can do in HYP_MODE you can't in SVC_MODE and the bootloader pretty much sets up all the key ones before it hands control to you. So I do it because it's safe and I am dumb, or is it not ARM savvy?

As we said you can go back up to it after you have done the setup of the registers in BOOT.S but then you will need to remember your in it and many of the registers you are playing with aren't directly shadowed like that to the secured modes that IRQ and FIQ will execute in.

There are only a couple of registers not shared between SVC_MODE and FIQ_MODE, IRQ_MODE the stack pointer being one you were needing to set so it's a lot easier to not make mistakes. As you found tracking register shadow issues down can be annoying to the MAX.
Makogan wrote: On any case thank you for helping me, I know it was my fault and that I wasted your time, I am sorry.
You didn't waste my time. I am working on a commercial product using the Pi3 and it is just part of time I have dedicated to learning the Pi3 anyhow.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Fri Jun 23, 2017 6:56 am

This thread and some questions/issues dealt with got me wondering. Specifically about aarch32 bit mode vs aarch64 bit mode, the connection to this is not on the what mode/level do we start but also what about the instruction set are they dropping us down from aarch64 to aarch32 if so how, is similar to getting out of HYP. Well no surprise there is a signal on the core sampled at reset which starts the core in aarch32 or aarch64 mode. There is text that talks about switching modes, most of the story is when you go up a level you can go to 64 but cant go up and from 64 to 32. Going down a level you can go from 64 to 32. (or in either case you can stay the same). But then you google and read forums and folks will say you cant get from 32 to 64 once you go to 32, but that doesnt appear to be true. The docs in this area talk about 32 bit EL3, but if you start 64 and can only get to 32 on the way down that would be at best EL2. I assume that means you used the strap on the core to boot 32 bit not 64 (the config.txt setting no doubt gets us this boot 32/64 as when you set it it is looking for aarch32 instructions)(arm_control=0x200). The RMR_ELx registers in theory will do a reset and allow you to switch modes, so I tried using them to do a warm reset and stay in the same mode. Didnt work, assumed the program would reset and print the text it prints on boot. nada. so perhaps it isnt that simple or it falls into the not implemented in our case, which is a possibility.

So booting aarch32 is easy with the config.txt settings, and sorting out the other cores and parking them is easy as well, curious to know what the boot aarch64 and switch to aarch32 mode looks like or what the working options are there as to get that done. (purely for educational purposes)

And booting from 0x0000 in aarch32 mode might have also helped you here (or 64 bit mode, doing some stuff then dropping down to 32, or 64 bit mode and just staying in aarch64 mode).

David

LdB
Posts: 1209
Joined: Wed Dec 07, 2016 2:29 pm

Re: Interrupts time and memory?

Fri Jun 23, 2017 8:16 am

So you are trying to make a thunk32 for the ARM8 in 64bit mode :-).

Yeah I am with you I would have thought it was possible. I might have a play I have a fairly decent array of code on 64bit mode.

ARM doesn't say there is a restriction
Image
It just says the obvious you can't host a 64 bit inside a 32bit and you can't host a 64bit hyp_mode

Have you tried scanning the linux ARM repository for the word "thunk"?
Martin Weidmann,ARM wrote:Take the example of a 64-bit OS running a 32-bit app. You are initially in EL1 as AArch64, running kernel code. The OS loads the context for the 32-bit app, and performs an exception return down to EL0. Switching to AArch32. On the next exception (e.g. scheduler tick, or system call) you transition back to EL1/AArch64.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Fri Jun 23, 2017 5:00 pm

I have a lot more research to do, first and foremost if I control the boot, what ELx level do we come in at, how to figure that out, how to change levels, and then how to switch modes when changing levels, etc. Now that I think about it the function I used to perform that system register write, I may have simply returned after, and didnt have any more output, maybe the program did something and didnt actually do the reset.

Anyway, dont know any of these answers yet other than the strap on the edge of the core that is one way to pick modes, and in that case the code the gpu places is aarch32, so the cpu is booting up aarch32 (or there is aarch64 code that is overwritten, by aarch32 code and then they do a warm reset). Need to see if there is a register where you can see status indicating the core strap was used to boot aarch32.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Fri Jun 23, 2017 5:03 pm

This is the wrong thread for this anyway should do some more experiments then make a thread to ask this question...Unless this topic has been covered (not using the config.txt file).

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Mon Jun 26, 2017 5:44 pm

What is PERIPHBASE on the pi3 chip? and/or where does the GIC address space start?

LdB
Posts: 1209
Joined: Wed Dec 07, 2016 2:29 pm

Re: Interrupts time and memory?

Mon Jun 26, 2017 6:35 pm

I would just read it out of the CBAR register
http://infocenter.arm.com/help/index.js ... EBDAI.html

Just tried it after bootloader start and is reading 0x00000000

I assume that is why it's sampled there from the GIC so you can find the sucker but they can disable it
http://infocenter.arm.com/help/index.js ... 01s01.html

Got an external GICv3 or GICv4 distributor component handy :-)

I am pretty sure U-Boot plays around with GIC so I would look there David.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Mon Jun 26, 2017 8:06 pm

Code: Select all

.globl cbar
cbar:
    //mrs x0,cbar_el1
    mrs x0,s3_1_c15_c3_0
    ret
I get zero as well, I am wondering if I have been through this exercise. usually there is more than one thing at PERIPHBASE to not set it to something, but perhaps the GIC is it...There is supposed to be a timer as well...The GIC was not listed in the optional section of the doc so I didnt think it was optional, but your finding makes a lot of sense all the way around, so as you probably already know I guess the stock IRQ/FIQ if prior cores just feeds in and interrupts in a pi3? I will find out shortly (I probably already did this but am going through the exercise again)..

David

User avatar
Ultibo
Posts: 158
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: Interrupts time and memory?

Tue Jun 27, 2017 12:31 am

dwelch67 wrote: but perhaps the GIC is it...There is supposed to be a timer as well...
David,

Neither the Pi2 or Pi3 contain a GIC, only the proprietary interrupt controller documented in the BCM2835 Peripherals document plus some extra extensions in the QA7 document for a small number of per CPU interrupts.

They do contain the Generic Timer device which is accessible via the mrc/mcr/mrrc instructions but apparently not via the memory mapped registers.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

LdB
Posts: 1209
Joined: Wed Dec 07, 2016 2:29 pm

Re: Interrupts time and memory?

Tue Jun 27, 2017 3:39 am

I read it the same way as David that it wasn't an optional on the CortexA-53 you had to have a GICv3/4 or GICv2 with restrictions. What you can do is use those GIC controller customize process to create and put registers wherever you wanted via it's virtualization process. So I assumed they mimic the old Pi1 controller with the GIC.

An example where a designer has used the GIC to virtualize classic processors irq systems
https://community.cadence.com/cadence_b ... ture-howto

Don't we love playing secret squirrels with hardware :-)

I now have an ARM account I will ask if you can have proprietary irq controller on a CortexA-53 that should give us some insight.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Tue Jun 27, 2017 1:23 pm

Currently playing with the timer, appears so far to be 250Mhz/256 like the system timer. May or may not investigate more. prior cores (arm11 mpcore) there was a timer that was PERIPHBASE memory mapped along side the GIC. This one as mentioned is coprocessor accesses. Not surprised the GIC is defeated the way they wedged these cores in. Was working on an interrupt blinker for aarch64 (will mess with aarch32 later, did try to switch modes unsuccessfully, may try again some day) on the pi3, elementary stuff, but got into trouble by reading the manual and assuming there was a GIC I had to deal with...

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Tue Jun 27, 2017 7:16 pm

forgive me for continuing to venture down this tangent (actually I am eventually going to come around and visit timer based interrupts on the pi3 in aarch64 which is mostly related to this thread).

The SYSTIMERCLO timer and the cntpct_el0 timer are fairly close, but there is about a 2 count or so per second difference between them. I had assumed/decided that SYSTIMERCLO is the 250MHz timer / 256 experimentally but not with a high degree of accuracy. I may have to try this on a different pi3. It is very noticable, both are going to average out to that same rate, but if you read one, read the other, print repeat the SYSTIMERCLO is gaining at a consistent rate.

I dont know the clock tree in this device, is the 250 generated with a separate pll from the 750 or 1000, etc? that might account for it, esp if some boards gain one way others faster or slower or systimer is slower.

Thought this was very interesting...

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Tue Jun 27, 2017 7:46 pm

ID_PFR1_EL1 shows that GICDISABLE is high, disabled. as you already knew or suspected...

User avatar
Ultibo
Posts: 158
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: Interrupts time and memory?

Tue Jun 27, 2017 11:47 pm

dwelch67 wrote:Currently playing with the timer, appears so far to be 250Mhz/256 like the system timer
dwelch67 wrote:The SYSTIMERCLO timer and the cntpct_el0 timer are fairly close, but there is about a 2 count or so per second difference between them.
David,

The QA7 document indicates that the clock for the generic timer comes directly from the 19.2MHz oscillator.

I'm not sure I have ever seen anything that says what the source of the system timer is but if they are different that might explain the small variations.
dwelch67 wrote:ID_PFR1_EL1 shows that GICDISABLE is high, disabled. as you already knew or suspected...
That seems to fit with all of the other information available.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Wed Jun 28, 2017 12:43 am

various ways to get the same answer. I may have written it wrong 250Mhz/256 is 900 something khz not mega.

but yes the arm clock cntpct_el0 can be affected by the 0x40000008 register as described in the document you posted. Writing 0x80000000 results in 19.2Mhz the ref clock. Taking the 0x06AAAAAB value up the arm clock can catch or pass the system timer.

0x06AAABC9 makes the two much closer, some drift still. Anyway, fun, can bump the arm timer way up from 977khz to 19.2mhz would really like a full arm clock speed timer, but oh well, generally that just burns through to many bits for most use cases.

Thanks for the link have seen that doc before, but forgot about it or that there was stuff I was interested in on it.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Wed Jun 28, 2017 3:54 am

Okay so aarch64 interrupts (events) were not that painful. Thanks for that baremetal doc, helped get moving on a few things that were in the main docs (ARM and TRM) just not as obvious. Add an example in my new pi3 repo.

Still have questions, the ESR didnt make sense and WFI doesnt make sense, although this is the first time I have really used it. In theory all four cores are in WFI but someone is handling the interrupts, and when leaving wfi they should hit code I placed (then loop back in). if after the wfi I disable interrupts, I get a couple dozen interrupts then they stop.

Anyway, aarch64 and the p3 in general has a lot more pain involved with execution levels, we come in as EL3 so you cant go higher when the exception comes in, using other levels and modes makes it much more complicated. No real interest in messing with that (with respect to interrupts). May still wish to try to drop down to aarch32 and then back up...

LdB
Posts: 1209
Joined: Wed Dec 07, 2016 2:29 pm

Re: Interrupts time and memory?

Thu Jun 29, 2017 3:44 am

I told you before not to get me started about EL3 :-)

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Interrupts time and memory?

Fri Jun 30, 2017 2:41 am

Dare I ask...So what is your preferred execution level then?

Return to “Bare metal, Assembly language”