xyb
Posts: 6
Joined: Thu Apr 17, 2014 9:38 pm

Weird. foo(char* fmt, ...) works on EL2 but doesn't work on EL1.

Sat Nov 24, 2018 2:00 am

Hi,

I'm playing with Zoltan's tutorial "https://github.com/bztsrc/raspi3-tutori ... tualmemory", try to see how to switch from EL2 to EL1. I found a weird issue.

In the main() function, I add another function after uart_init() to print out "hello world":

Code: Select all

void hello(char* fmt)
{
    uart_puts(fmt);
}

void main()
{
    char *s="Writing through MMIO mapped in higher half!\n";

    // set up serial console
    uart_init();
    
    hello("Hello World!\n");
    
    // setup pagging
    ....
}
Every thing works fine. I can see "Hello World!" from the uart output. Then I change the hello function like this:

Code: Select all

void hello(char* fmt, ...)
{
    uart_puts(fmt);
}

void main()
{
    char *s="Writing through MMIO mapped in higher half!\n";

    // set up serial console
    uart_init();
    
    hello("Hello World!\n");
    
    // setup pagging
    ....
}
I can't see "Hello World!" now. And my RPi 3 seems be hung in hello(). The only difference is that I change the hello function signature from (char* fmt) to (char* fmt, ...).

Now I modify start.S "https://github.com/bztsrc/raspi3-tutori ... ry/start.S", don't switch to EL1, jump to main() from EL2 directly:

Code: Select all

    ....

    // set stack before our code
    ldr     x1, =_start
    mov sp, x1   // <---- skip to switch from EL2 to EL1, jump to main() directly
    bl main
 
    // set up EL1
    ....
Now I can see "Hello World!" again. At first, I guess it would be caused by that the stack pointer is messed up when we switch from EL2 to EL1, But when I print out the sp register value in main(), I found the values under the tow cases are the same.

Why (char* fmt, ...) doesn't work under EL1?

LdB
Posts: 1102
Joined: Wed Dec 07, 2016 2:29 pm

Re: Weird. foo(char* fmt, ...) works on EL2 but doesn't work on EL1.

Sat Nov 24, 2018 4:01 am

I initially said there were two errors with code on re-reading the C standard I am going to amend this to 1. It is valid
to not use a variadic argument and not have to tell the compiler it is a special case of passing variables.

Your code as shown shouldn't compile and throw a fatal error the fact the GCC compiler is allowing it is one of it's weakness
and it likely got itself confused.

1.) The "..." is a variadic and you have not told the compiler what standard that is to take.
Usually what is required is the library it is to take for that like #include <stdarg.h>
That standard tells it how that variadic is to be passed across the interface.

You then don't use the variadic so I guess the optimizer messed it up because it optimized away the variadic
in one half the code not the other and you sort of confirm that in what you say about the stack readings.

I am curious did GCC give any warnings at all or did you just ignore them?

Update: It is an interesting story actually looking at the definition of "..." across the various standards.
The fact it doesn't get used makes it really interesting to how it should be interpretted. I am actually going
to further amend my answer to it depends what version C you are compiling for as to what it should do with
this example and I am as confused as the compiler is. The only real answer would be if the compiler could
realize the variadic is not in use and optimize the whole lot away, which is what you are doing in predicting
that the result should be the same. Once you defined a use of the variadic it becomes clear what you are
doing but in this form it's problematic. The problem is the standards are built around using the variadic not
leaving it unused, there entire framing is about use.

xyb
Posts: 6
Joined: Thu Apr 17, 2014 9:38 pm

Re: Weird. foo(char* fmt, ...) works on EL2 but doesn't work on EL1.

Sat Nov 24, 2018 6:06 pm

I think I know the root cause.

I disassemble the code, found the following difference:

for hello(char* fmt) version, it is very simple:

Code: Select all

0000000000000000 <hello>:
   0:   a9be7bfd        stp     x29, x30, [sp, #-32]!
   4:   910003fd        mov     x29, sp
   8:   f9000fe0        str     x0, [sp, #24]
   c:   f9400fe0        ldr     x0, [sp, #24]
  10:   94000000        bl      0 <uart_puts>
  14:   d503201f        nop
  18:   a8c27bfd        ldp     x29, x30, [sp], #32
  1c:   d65f03c0        ret
for hello(char* fmt, ...) version, it is:

Code: Select all

0000000000000000 <hello>:
   0:   a9b27bfd        stp     x29, x30, [sp, #-224]!
   4:   910003fd        mov     x29, sp
   8:   f9000fe0        str     x0, [sp, #24]
   c:   f90057e1        str     x1, [sp, #168]
  10:   f9005be2        str     x2, [sp, #176]
  14:   f9005fe3        str     x3, [sp, #184]
  18:   f90063e4        str     x4, [sp, #192]
  1c:   f90067e5        str     x5, [sp, #200]
  20:   f9006be6        str     x6, [sp, #208]
  24:   f9006fe7        str     x7, [sp, #216]
  28:   3d800be0        str     q0, [sp, #32]
  2c:   3d800fe1        str     q1, [sp, #48]
  30:   3d8013e2        str     q2, [sp, #64]
  34:   3d8017e3        str     q3, [sp, #80]
  38:   3d801be4        str     q4, [sp, #96]
  3c:   3d801fe5        str     q5, [sp, #112]
  40:   3d8023e6        str     q6, [sp, #128]
  44:   3d8027e7        str     q7, [sp, #144]
  48:   f9400fe0        ldr     x0, [sp, #24]
  4c:   94000000        bl      0 <uart_puts>
  50:   d503201f        nop
  54:   a8ce7bfd        ldp     x29, x30, [sp], #224
  58:   d65f03c0        ret
The difference is that in variadic version, compiler save all registers in the stack. I think it is related to aarch64 ABI, because the number of the parameters are not determined, so compiler save all registers. Especially, it also saves registers q0~q7 which are used for SIMD instruction. After reading a lot of ARM documents, I believe the issue is caused on saving these SIMD registers.

From ARM documents, http://infocenter.arm.com/help/topic/co ... BGEAB.html, whether the application can access SIMD registers is controlled by FPEN bits of cpacr_el1 register. The default value of FPEN is 0b00 which means any accessing SIMD registers will generate an exception (be trapped). I dump the capcr_el1 register immediately after switching from el2 to el1, the value is 0, that means any instruction to save q0~q7 register generated compiler are trapped. So hello(fmt, ...) doesn't work, but hello(fmt) works. Because hello(fmt) doesn't have instructions to access SIMD registers.

After understanding the root cause, I modify the start.s, add the following instructions to disable this trap:

Code: Select all

mov x19, 3 << 20     // Set 0b11 on FPEN bits of cpacr_el1 to disable trap for accessing SIMD registers
msr cpacr_el1, x19
Now, hello(fmt, ...) works either. So the conclusion is if you want to use variadic function, like printf(...), you should process corresponding traps or disable them.

Thanks LdB! Your replay also give me a lot of hints. Appreciate!

bzt
Posts: 343
Joined: Sat Oct 14, 2017 9:57 pm

Re: Weird. foo(char* fmt, ...) works on EL2 but doesn't work on EL1.

Sun Nov 25, 2018 10:48 am

xyb wrote:
Sat Nov 24, 2018 6:06 pm
Now, hello(fmt, ...) works either. So the conclusion is if you want to use variadic function, like printf(...), you should process corresponding traps or disable them.

Thanks LdB! Your replay also give me a lot of hints. Appreciate!
Well done! I can confirm that I ran into the same problem with vaargs, read the readme for 12_printf tutorial. And I solved that by enabling SIMD just as you did. It was a very clever debugging on your part!

Cheers,
bzt

xyb
Posts: 6
Joined: Thu Apr 17, 2014 9:38 pm

Re: Weird. foo(char* fmt, ...) works on EL2 but doesn't work on EL1.

Sun Nov 25, 2018 3:15 pm

bzt wrote:
Sun Nov 25, 2018 10:48 am

Well done! I can confirm that I ran into the same problem with vaargs, read the readme for 12_printf tutorial. And I solved that by enabling SIMD just as you did. It was a very clever debugging on your part!

Cheers,
bzt
Nice tutorial! If I read your tutorial earlier, I would save a lot of hours. ;)

User avatar
Paeryn
Posts: 2516
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: Weird. foo(char* fmt, ...) works on EL2 but doesn't work on EL1.

Sun Nov 25, 2018 5:35 pm

I'm surprised that GCC is pushing the FP registers onto the stack, AAPCS says when calling variadic functions all parameters are passed using integer registers and stack only.

<Addendum>
Ah, AAPCS64 does things differently, it states that variadic functions puts the passed argument registers onto the stack in a certain way (with integer and fp registers having separate blocks). It allows the callee to not store any FP/NEON registers so long as it never accepts FP/NEON types. Looks like GCC hadn't made that optimisation.
She who travels light — forgot something.

Return to “Bare metal, Assembly language”