romell
Posts: 25
Joined: Mon Jul 23, 2012 6:57 pm

Virtual memory and linking

Tue Jul 31, 2012 2:04 pm

Hi,

I have some, mostly theoretical issues when setting up my virtual memory system. What I want is a single binary (the kernel) to be loaded into continuous physical memory starting at 0x00008000. Only a very small part of the code that is responsible for setting up an initial page table and activating the MMU should be linked to run in this address space. The rest of the code should live somewhere in the top half of the virtual memory, leaving room for user space processes at the bottom.

Is this possible or do I need to rethink it? Another possibility is of course a two step process where I first set up the virtual memory and then load the main kernel from the SD card, but I would really like to avoid that...

User avatar
Cycl0ne
Posts: 102
Joined: Mon Jun 25, 2012 8:03 am

Re: Virtual memory and linking

Tue Jul 31, 2012 3:51 pm

you have to get yourself some infos on OS Design.
http://wiki.osdev.org
is a good place to start.

Linux, Windows, MacOSx, all compile their binaries (programms) to 0x0. so if you put your kernel at 0x0800, you have to write your own compiler/linker tools.
Most 32 bit OS tend to use a 2:2 approach (2gig user, 2gig system, so that all user calls to system will end up at an adress bigger 0x8000 0000. this makes it easier (and faster) for system calls. Negative effect: you program can only use 2gig adress space. OSX tends to use 4:4 allocation, but this brings the look aside buffer into big trouble, cause every user <-> system call you need to refresh the whole virtual adress mapping.

So you should first read the link above and think about your proposal.

in my OS design all is flat. i dont use virtual adress space. thus i have the problem to relocate the binary files when loading them into ram. :-( but every concept has its pros and cons.

romell
Posts: 25
Joined: Mon Jul 23, 2012 6:57 pm

Re: Virtual memory and linking

Tue Jul 31, 2012 5:08 pm

Thanks for the link, I will check it out.

Maybe I was a bit unclear about the placement of the kernel. I don't it placed at virtual address 0x8000 and it doesn't really matter where its placed in physical memory as long as I can boot into it and set up an initial page table (identity mapping). My problem is more related to how I can instruct the linker to make parts of the code to be executed in the upper part of the address space, while still being loaded into the lower part together with the other code.

Sorry for the messy explanation, I will try to summarize my intended program flow:

* The whole binary gets loaded into physical address 0x8000 with MMU turned off. (Just like any other bare metal program)
* A small assembly routine sets up a temporary page table with an identity mapping to itself and a virtual-to-physical mapping for the remaining part of the kernel, e.g. 0xA0000000->[main kernel entry point]
* Turn on MMU and jump to 0xA0000000.

This is where I expect strange things to happen if the program is linked using absoute addresses starting from 0x8000...

I hope this makes at least some sense. Any ideas?

Velko
Posts: 7
Joined: Mon Feb 20, 2012 9:50 pm

Re: Virtual memory and linking

Tue Jul 31, 2012 7:52 pm

Well, it makes perfect sense to me.

I've been playing around with similar setup on both x86 and ARM (qemu-system-arm, haven't had time to try it on actual RasPi). All you need is slightly more sophisticated linker script. Something like this (you may notice that I prefer 3:1 userspace/kernel split):

Code: Select all

ENTRY (_start)
load_start = 0x8000;
virt_start = 0xC0000000;
SECTIONS
{
    . = load_start;
    .lowtext :
    {
        *(.lowtext)
    }
    . = . + virt_start;
    .text : AT(code_start - virt_start) 
    { 
        code_start = .;
        *(.text*)
    }
    /* other higher half sections goes here */
}
You write all the code meant to run linked at 0x8000 in separate sections .lowtext, the rest goes into regular .text. When linked together, there will be 2 sections, continuous by physical addresses, but "higher-half" .text will start at 0xC0000000 + 0x8000 + whatever the size of .lowtext is. Having it this way, instead of forcing .text to start at "round" address (0xC0000000 in this case) makes it much, much easier to set up the initial page tables.

As long as you do not do function calls between these 2 sections (except for that one special jump to higher-half), you'll be fine.

And setting up initial page table needs not to be pure assembly. You can use C for it as well, using GCC attributes __section__ to compile functions into .lowtext section.

romell
Posts: 25
Joined: Mon Jul 23, 2012 6:57 pm

Re: Virtual memory and linking

Tue Jul 31, 2012 8:51 pm

Thank you Velko, that was exactly what I was looking for! I will try it out tomorrow. Too many beers for any decent coding now I'm afraid. :)

dwelch67
Posts: 954
Joined: Sat May 26, 2012 5:32 pm

Re: Virtual memory and linking

Wed Aug 01, 2012 1:27 am

http://github.com/dwelch67/raspberrypi/ the extest example starts by coming up bare metal and sets up a minimal mmu table and enables the mmu. I matched real and virtual but just as easy to make the virtual somewhere else. get the ARM ARM and ARM TRM for the processor core.

romell
Posts: 25
Joined: Mon Jul 23, 2012 6:57 pm

Re: Virtual memory and linking

Fri Aug 03, 2012 11:42 pm

It works! :D My kernel is now running in the upper parts of virtual memory. I'm currently using a 3:1 split between user and kernel space, but I will probably change that to 1:1 since there seems to be a nice hardware support for that kind of split (see section 6.12.1 of the ARM TRM).

The linker script provided by Velko sort of worked. I had to remove the "AT()" command to stop the linker from spitting out a 1GB+ binary..

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sun May 19, 2019 10:36 am

@romell, what did you do about exception vector table?

Currently I have exception table setup at 0x0. Can we change the exception table base address ?
Let there be some light .../\...

LdB
Posts: 1143
Joined: Wed Dec 07, 2016 2:29 pm

Re: Virtual memory and linking

Sun May 19, 2019 11:55 am

VBAR register sets it anywhere you like on a 4K byte aligned boundary, reset value is 0x0

I assume you are after EL1
http://infocenter.arm.com/help/index.js ... ECBDH.html

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22701
Joined: Sat Jul 30, 2011 7:41 pm

Re: Virtual memory and linking

Sun May 19, 2019 3:36 pm

zeoneo wrote:
Sun May 19, 2019 10:36 am
@romell, what did you do about exception vector table?

Currently I have exception table setup at 0x0. Can we change the exception table base address ?
Wow, that's a nearly 7 year thread necro. A new record I think.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

fruitoftheloom
Posts: 19781
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Virtual memory and linking

Sun May 19, 2019 4:15 pm

zeoneo wrote:
Sun May 19, 2019 10:36 am
@romell, what did you do about exception vector table?

Currently I have exception table setup at 0x0. Can we change the exception table base address ?

Bit late to ask ?
romell
Joined: Mon Jul 23, 2012 7:57 pm
Last active: Sat Nov 01, 2014 12:25 am
adieu

My other Computer is an Asus CS10 ChromeBit running Chrome Operating System.
HP Envy 4500 Wireless Printer supported by HPLIP software in Raspbian Buster.
Raspberry Pi Model 2B v1.1

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Mon May 20, 2019 5:08 am

LdB wrote:
Sun May 19, 2019 11:55 am
VBAR register sets it anywhere you like on a 4K byte aligned boundary, reset value is 0x0

I assume you are after EL1
http://infocenter.arm.com/help/index.js ... ECBDH.html
Yeah that's what I was exactly looking for. Thanks Ldb.
jamesh wrote: Wow, that's a nearly 7 year thread necro. A new record I think.
I will break few more records here. :D

Two things I love about raspberrypi are
1. Raspberry Pi itself
2. Community folks who are open to help people out.

Thank you guys.
Let there be some light .../\...

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sat Jul 13, 2019 4:17 am

Guys I am trying implement higher half kernel, but I think I am stuck in chicken egg problem.

AFAK ttbr0 and ttbr1 needs physical addresses. I am using following linker script to link boot code at 0x8000 and kernel code at 2GB.


I don't know how to find out the physical address of labels
__first_lvl_tbl_base, __second_lvl_tbl_base
I have deliberately put page table labels at the end because I don't want to increase the size of the binary kernel loaded at 0x8000.

My plan is to initialize page tables by mapping physical address of kernel code '__code_start' to 2GB (linked address), Again here I will need to find the physical address of label '__code_start'

Any help is appreciated.

Code: Select all

ENTRY(_start)

/* Address where boot section will be linked */
load_start = 0x8000;

/*
 *
 *  Address where kernel will be linked
 *  We will initialize paging and map the kernel physical address to
 *  virt_start address. And then jump to kernel code.
*/
virt_start = 0x80000000; /* 2GB */

SECTIONS
{
    /* Starts at LOADER_ADDR. */
    . = 0x8000;
    __start = .;
    __text_boot_start = .;
    .text.boot :
    {
        KEEP(*(.text.boot))
    }
    __text_boot_end = .;
    . = ALIGN(4096);
    __text_boot_end_aligned = .;
    /* Kernel sections are placed after this */
    . = . + virt_start;

	.text : AT(__code_start - virt_start) { 
        __code_start = .;
        *(.text*)
    }
	.rodata : { *(.rodata) }
	.data : { *(.data) }

    __bss_start = .;
    .bss :
    {
        bss = .;
        *(.bss*)
    }
    __bss_end = .;

    . = . + 131072; /* 128KB for svc stack */
    __svc_stack_base = .;

    . = . + 1024; /* 1024 Bytes for irq stack */
    __irq_stack_base = .;

    . = ALIGN(16384); /* align 16KB for page tables */

    __first_lvl_tbl_base = .;
    . = . + 4096; /* First level page table size */
    __first_lvl_tbl_end = .;

    /* 1KB for each 2nd lvl page table = 4 Bytes * 256 */
    /* How many 2nd lvl tables ? */
    __second_lvl_tbl_base = .;
    . = . + 0x00100000; /* 1MB for 2nd level page tables */
    __second_lvl_tbl_end = .;

    __kernel_end = .;
    __kernel_size = __kernel_end - __code_start;

}


I was referring to this tutorial which is using 64Bit architecture. https://github.com/s-matyukevich/raspbe ... /rpi-os.md

I don't understand why does it work, the guy linked all the code at 0xffff000000000000 even boot section and later to find out physical address of pgdir label he just used

Code: Select all

 adrp    x0, pg_dir
which should give linked address which is not even accessible.

I read about the PC relative addressing when I tried using

Code: Select all

ldr r0, =label
it gives me linked address.
Let there be some light .../\...

LdB
Posts: 1143
Joined: Wed Dec 07, 2016 2:29 pm

Re: Virtual memory and linking

Sat Jul 13, 2019 7:38 am

I can't make sense of some of what or why you want to do some things but i will answer what I can

ttbr0 and ttbr1 have to be physical addresses they are the start point for locating the virtual tables.

At assembler level the address of __code_start is simply something like

Code: Select all

ldr r0, =__code_start
I am guessing you already clear the BSS with __bss_start so not sure why that is causing you trouble.

This number is clearly going to be negative and as far as I can tell non-sensical

Code: Select all

AT(__code_start - virt_start)
virt address is 0x80000000, and _code_start has to be less than 0x3F000000 the last address possible.

Again I don't understand why you are doing all the linker stuff you can just compile the kernel with GCC -fPIC option
PIC stands for "Position Independent Code" and you can put the kernel wherever you want

At a deeper level I can't work out why you even care where the kernel is if you are going to run proper virtualization.
Your apps or tasks or whatever are going to be running in a virtual memory address.
As some posters noted usually all the none virtual code is squeezed together somewhere typically down the bottom at 0x0.
Some of the posters I can see they want to run flat memory so I get why they position the kernel but you aren't going that path.

So Questions:
You seem to be trying to make an image file with a physical memory block and a virtual memory block. Even if you could
make such a file how do you propose the VC4 when it picks that file up it would load the virtual memory block given the MMU is off?
The alternative to that is this is an elf file and you have a second stage loader you haven't told use about?
From what I read you aren't so can I get an explaination of why you care where the kernel is?
Why can't you simply copy the kernel to where you want it (David Welch gave example), why does it have to load there?

PS: You asked how the guy doing the O/S worked it
He create a file with one section starting supposedly at . = 0xffff000000000000 that will become true only when he turns the MMU on.
The VC4 loads the file at 0x0 because uses kernel_old=1 on his config.txt (https://github.com/s-matyukevich/raspbe ... config.txt)
The VC4 loader is dumb it just knows to load the file as is in either position 0x0 or 0x8000 or 0x80000(AARCH64)
So he knows the start of his file will start as 0x0 and he needs it become 0xffff000000000000 when he turns the MMU on.
IN SHORT he faked it all don't take the address literally, they become true only if the MMU is setup and turned on correctly.

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sat Jul 13, 2019 10:37 am

LdB wrote: virt address is 0x80000000, and _code_start has to be less than 0x3F000000 the last address possible.
I don't think it's correct. Checkout the kernel map file here (80009000 T __code_start)

Code: Select all

0000804c t .NotInHypMode
8001a5ac t .udivsi3_skip_div0_test
00000040 a ARM_F_BIT
00000080 a ARM_I_BIT
00000011 a ARM_MODE_FIQ
0000001a a ARM_MODE_HYP
00000012 a ARM_MODE_IRQ
0000001f a ARM_MODE_MASK
00000013 a ARM_MODE_SVC
0000001f a ARM_MODE_SYS
0000001b a ARM_MODE_UND
00000010 a ARM_MODE_USR
800090c8 T ARMaddrToGPUaddr
80031920 B Allocations
8001b8ec r BitFont
8001292c T BitGetSigned
80012828 T BitGetUnsigned
80012570 T BitSet
00000040 a CPSR_FIQ_INHIBIT
00000080 a CPSR_IRQ_INHIBIT
00000017 a CPSR_MODE_ABORT
00000011 a CPSR_MODE_FIQ
00000012 a CPSR_MODE_IRQ
00000013 a CPSR_MODE_SVR
0000001f a CPSR_MODE_SYSTEM
0000001b a CPSR_MODE_UNDEFINED
00000010 a CPSR_MODE_USER
00000020 a CPSR_THUMB
800184c4 t ClearArea16
8000a99c T ClearReg
80020c68 D ConfigurationDescriptor
800161a8 T CopyUnAlignedString
80029788 B Core
80029910 B CorePhysical
80020c84 D DeviceDescriptor
80033160 B Devices
80020c24 D FirstAllocation
80029794 B FirstFreeAllocation
80009008 T GET32
800090d4 T GPUaddrToARMaddr
8000bbf8 T HcdChannelInterruptToError
8000c2f4 T HcdChannelSendInterruptPoll
8000bda8 T HcdChannelSendInterruptPollOne
8000cd3c T HcdChannelSendWait
8000c758 T HcdChannelSendWaitOne
8000aa28 T HcdInitialize
8000c53c T HcdInterruptPoll
8000b788 T HcdPrepareChannel
800109ac T HcdProcessRootHubMessage
8000adfc T HcdReceiveFifoFlush
8000a8d4 T HcdReset
8000b730 T HcdStart
8000d328 T HcdSumbitControlMessage
8000cf84 T HcdSumbitInterruptOutMessage
8000baf0 T HcdTransmitChannel
8000acec T HcdTransmitFifoFlush
80029920 B Heap
800120ec T HidAttach
80011b38 T HidDeallocate
8001109c T HidDetached
800116e8 T HidEnumerateActionAddField
800110b8 T HidEnumerateActionCountField
800111c0 T HidEnumerateActionCountReport
80011d14 T HidEnumerateReport
80012cb4 T HidGetFieldValue
800126a4 T HidGetReport
80011c28 T HidLoad
80011e50 T HidParseReportDescriptor
80012954 T HidReadDevice
80011c68 T HidSetProtocol
80012780 T HidSetReport
800331e0 B HidUsageAttach
80012b34 T HidWriteDevice
80029784 B Host
80029914 B HostPhysical
80010480 T HubAttach
8000fa38 T HubChangeFeature
8000fadc T HubChangePortFeature
80010128 T HubCheckConnection
8001040c T HubCheckConnectionDevice
80010380 T HubCheckForChange
8000f5b0 T HubChildDetached
8000fe40 T HubChildReset
8000f658 T HubDeallocate
80020c28 D HubDescriptor
8000f5e8 T HubDetached
8000f858 T HubGetStatus
8000f6f8 T HubLoad
8000fe80 T HubPortConnectionChanged
8000f940 T HubPortGetStatus
8000fc7c T HubPortReset
8000fb90 T HubPowerOn
8000f738 T HubReadDescriptor
80033120 B InterfaceClassAttach
800133f0 T KbdLoad
800134f4 T KeyWasDown
8001391c T KeyboadGetKeyIsDown
80012d84 T KeyboardAttach
80012d38 T KeyboardDeallocate
80012cc4 T KeyboardDetached
80013538 T KeyboardGetAddress
80013a54 T KeyboardGetChar
800139a0 T KeyboardGetKeyDown
800138d8 T KeyboardGetKeyDownCount
8001368c T KeyboardGetLedSupport
80013894 T KeyboardGetModifiers
8001347c T KeyboardIndex
800136d0 T KeyboardPoll
8001359c T KeyboardSetLeds
80013a04 T KeyboardUpdate
8000d770 T MemoryAllocate
8000da44 T MemoryCopy
8000d9a4 T MemoryDeallocate
8000d768 T MemoryReserve
80009e6c T MicroDelay
80013ba4 T MouseAttach
80013fd4 T MouseCount
80013b58 T MouseDeallocate
80013ae4 T MouseDetached
80013f70 T MouseGetAddress
8001427c T MouseGetButtonIsPressed
80014238 T MouseGetButtons
800141ec T MouseGetPosition
80014120 T MouseGetPositionX
80014164 T MouseGetPositionY
800141a8 T MouseGetWheel
80013ef8 T MouseIndex
80013e6c T MouseLoad
80013ff4 T MousePoll
80020bf0 r PIo2
80009000 T PUT32
8002978c B PhyInitialised
8000da90 T PlatformLoad
80029780 B Power
80029918 B PowerPhysical
800090e0 T RPI_GetArmTimer
80009118 T RPI_GetIrqController
800094f4 T RPI_Mailbox0Read
800094c8 T RPI_Mailbox0Write
8000956c T RPI_PropertyAddTag
80009bb8 T RPI_PropertyGet
80009534 T RPI_PropertyInit
80009b6c T RPI_PropertyProcess
8000a7ec T ReadBackReg
800160b0 t ReadLFNEntry
800297e8 B RootHubDeviceNumber
00000800 a SCTLR_ENABLE_BRANCH_PREDICTION
00000004 a SCTLR_ENABLE_DATA_CACHE
00001000 a SCTLR_ENABLE_INSTRUCTION_CACHE
800213e4 d SD_TYPE_NAME
8000ac38 T SetReg
80020c64 D String0
80020c34 D String1
80029798 b TreeLevelInUse
8000ebe4 T UsbAllocateDevice
8000efe0 T UsbAttachDevice
8000f4d4 T UsbAttachRootHub
8000f57c T UsbCheckForChange
8000e550 T UsbConfigure
8000e168 T UsbControlMessage
8000ecdc T UsbDeallocateDevice
8000dae8 T UsbGetDescription
8000e320 T UsbGetDescriptor
8000dac8 T UsbGetRootHub
8000edb4 T UsbGetString
8000eb78 T UsbInitialise
8000e848 T UsbReadDeviceDescriptor
8000ee8c T UsbReadString
8000ee14 T UsbReadStringLang
8000ea70 T UsbSetAddress
8000e46c T UsbSetConfiguration
8000e974 T UsbSetInterface
8000df70 T UsbShowTree
80018564 t WriteChar16
8000a7e4 T WriteThroughReg
8000a71c T WriteThroughRegMask
800213d8 d _GLOBAL_OFFSET_TABLE_
8001a6d0 T __adddf3
8001acd0 T __addsf3
8001ab94 T __aeabi_cdcmpeq
8001ab94 T __aeabi_cdcmple
8001ab78 T __aeabi_cdrcmple
8001b2ec T __aeabi_cfcmpeq
8001b2ec T __aeabi_cfcmple
8001b2dc T __aeabi_cfrcmple
8001ac24 T __aeabi_d2f
8001a6d0 T __aeabi_dadd
8001abac T __aeabi_dcmpeq
8001abf4 T __aeabi_dcmpge
8001ac0c T __aeabi_dcmpgt
8001abdc T __aeabi_dcmple
8001abc4 T __aeabi_dcmplt
8001a6c4 T __aeabi_drsub
8001a6cc T __aeabi_dsub
8001aa2c T __aeabi_f2d
8001b37c T __aeabi_f2iz
8001b3d8 T __aeabi_f2uiz
8001acd0 T __aeabi_fadd
8001b304 T __aeabi_fcmpeq
8001b34c T __aeabi_fcmpge
8001b364 T __aeabi_fcmpgt
8001b334 T __aeabi_fcmple
8001b31c T __aeabi_fcmplt
8001b108 T __aeabi_fdiv
8001af70 T __aeabi_fmul
8001acc4 T __aeabi_frsub
8001accc T __aeabi_fsub
8001aa04 T __aeabi_i2d
8001ae94 T __aeabi_i2f
8001a6c0 W __aeabi_idiv0
8001aa80 T __aeabi_l2d
8001aec4 T __aeabi_l2f
8001a6c0 W __aeabi_ldiv0
8001a9e0 T __aeabi_ui2d
8001ae8c T __aeabi_ui2f
8001a5ac T __aeabi_uidiv
8001a6a0 T __aeabi_uidivmod
8001aa6c T __aeabi_ul2d
8001aeb4 T __aeabi_ul2f
80033228 B __bss_end
800212e5 D __bss_start
8001aaf0 T __cmpdf2
8001b278 T __cmpsf2
80009000 T __code_start
000080f0 t __create_page_tables
8000a17c t __delay_53
8000a18c t __delay_70
8001b108 T __divsf3
8001aaf0 T __eqdf2
8001b278 T __eqsf2
8001aa2c T __extendsfdf2
80054000 B __first_lvl_tbl_base
80055000 B __first_lvl_tbl_end
8001b37c T __fixsfsi
8001b3d8 T __fixunssfsi
8001aa80 T __floatdidf
8001aec4 T __floatdisf
8001aa04 T __floatsidf
8001ae94 T __floatsisf
8001aa6c T __floatundidf
8001aeb4 T __floatundisf
8001a9e0 T __floatunsidf
8001ae8c T __floatunsisf
8001aae0 T __gedf2
8001b268 T __gesf2
8001aae0 T __gtdf2
8001b268 T __gtsf2
80019278 T __ieee754_rem_pio2f
80053628 B __irq_stack_base
800196d8 T __kernel_cosf
80155000 B __kernel_end
800198f4 T __kernel_rem_pio2f
8001a258 T __kernel_sinf
0014c000 A __kernel_size
8001aae8 T __ledf2
8001b270 T __lesf2
8001aae8 T __ltdf2
8001b270 T __ltsf2
8001af70 T __mulsf3
8001aaf0 T __nedf2
8001b278 T __nesf2
80055000 B __second_lvl_tbl_base
80155000 B __second_lvl_tbl_end
00008000 T __start
8001a6cc T __subdf3
8001accc T __subsf3
80053228 B __svc_stack_base
000081dc T __text_boot_end
00009000 T __text_boot_end_aligned
00008000 T __text_boot_start
8001ac24 T __truncdfsf2
8001a5ac T __udivsi3
000080fc t _continue_map
80018a38 T _doprnt
80009030 T _enable_interrupts
         U _exception_table
80009024 T _get_stack_pointer
000081a8 t _inf_loop
00008000 T _start
80029760 b all_pages_array
80009f40 T alloc_page
80029790 B allocated
800298cc B angle
80009e98 T append_page_list
80029764 b arm_mem_size
800090ec T arm_timer_init
80016078 T ascii_for_utf_16
80021400 B bss
80009454 T bzero
00008080 t clear_bss
80021520 b clearers
800298d0 B console
8001868c T console_putchar
8001875c T console_puts
8000a31c T copy
8001a59c T copysignf
80021054 D cosTheta
800191b4 T cosf
80021640 B count_irqs
800298a8 b current_sd_partition
8000918c T data_abort_vector
80009070 T data_abort_vector_asm
8002977c B databuffer
8001c8fc r digits_lc
8001c8ec r digits_uc
80020c1c D dmaChNum
8000a280 t dma_0_irq_clearer
8000a2c0 t dma_0_irq_handler
8002976c B dma_cb_page
80029774 B dma_dest_page_1
80029770 B dma_dest_page_2
80029778 B dma_src_page_1
800090bc T dmb
80016ad4 T doNothingFat16
80016ad0 T doNothingFat32
80016d5c T do_rotate
80009010 T dummy
80018470 T emit_float
800183ec T emit_uint16_t
8001841c T emit_uint32_t
800183d8 T emit_uint8_t
80009014 T enable_irq
80021058 D fShaderStr
8001a3a0 T fabsf
800091a8 T fast_interrupt_vector
80021040 D file_to_search
800163c0 T find_file_in_directory
8001a3a8 T floorf
800189e4 T fputc
80009fac T free_page
80029904 B free_pages
800161d0 T getFatEntrySectorAndOffset
80016224 T getFirstSectorOfCluster
8001884c T get_console_frame_buffer
80018784 T get_console_width_height_depth
80016424 T get_file_size
80009ff0 T get_mem_size
80015fac T get_next_dir_name
80009f24 T get_num_of_free_pages
80021400 b handlers
8000aea4 T hcd_start
8000a240 T hexstrings
80020be4 r init_jk
80016b5c T init_v3d
80015cac T initialize_fat
80009460 T interrupts_init
800091c4 T irq_handler
80009040 T irq_handler_asm_wrapper
80015ff8 T is_same_file
8001897c T kernel_main
80029804 B keyboardAddresses
80029814 B keyboardCount
80033208 B keyboards
8001b6ec r keymap
00008000 A load_start
8000a040 T mem_init
800189c0 T memcpy
80009430 T memset
80033218 B mice
80029818 B mouseAddresses
80029828 B mouseCount
80009f1c T next_page_list
8002084c r npio2_hw
80029768 b num_pages
800297ec B old_keys_down
80009ee0 T peek_page_list
80009ee8 T pop_page_list
8000a6a0 T power_on_host_controller
80009170 T prefetch_abort_vector
8000a2d0 T print_area
80016858 T print_directory_cluster_contents
800168a8 T print_directory_contents
80016980 T print_entries_in_sector
800166cc T print_file_cluster_contents
8001683c T print_root_directory_info
8000a630 T print_usb_power_state
800189fc T printf
80029654 b property.4215
80021650 b pt
80029650 b pt_index
80009ebc T push_page_list
800090b4 T read_cpu_id
80016780 T read_file
800092ac T register_irq_handler
80009124 T reset_vector
8001a47c T scalbnf
8001539c t sdAppSendOpCond
80029830 b sdCard
80015a6c T sdCardCSD
80015b50 T sdClearBlocks
80020c98 d sdCommandTable
80014318 t sdDebugResponse
800156d0 T sdInitCard
8001517c t sdResetCard
8001505c t sdSendAppCommand
800150d8 t sdSendCommand
800152e4 t sdSendCommandA
80014900 t sdSendCommandP
800145d4 t sdSetClock
80015460 t sdTransferBlocks.part.5
800144d8 t sdWaitForData
80014390 t sdWaitForInterrupt
80015a8c T sdcard_read
80015aec T sdcard_write
80016244 T search_directory_cluster_contents
800162bc T search_directory_contents
80016674 T search_entries_in_sector
80016458 t search_entries_in_sector.part.1
800298c8 b selected_fat_type
80016ad8 T set_max_speed
8000a35c T show_dma_demo
80009f14 T size_page_list
80009158 T software_interrupt_vector
80019184 T strlen
80016df0 T test_triangle
80009e60 T tick_difference
80009dd8 T timer_getTickCount32
80009df0 T timer_getTickCount64
80009c68 T timer_init
8002975c b timer_regs
80009c94 T timer_set
8000a5a0 T turn_off_usb
8000a5e8 T turn_on_usb
800208cc r two_over_pi
8000a1f0 T uart_getc
8000a158 T uart_init
8000a1cc T uart_putc
8000a218 T uart_puts
80009cb0 T udelay
80009140 T undefined_instruction_vector
80009368 T unregister_irq_handler
80020c20 D usb_hcd_device_id
80017de0 T v3d_AddShadderToScene
800179c0 T v3d_AddVertexesToScene
800178f0 T v3d_InitializeScene
800182a0 T v3d_RenderScene
800180d8 T v3d_SetupBinningConfig
80017eec T v3d_SetupRenderControl
80016ca0 T v3d_execute_code
80016d00 T v3d_execute_qpu
80016bb0 T v3d_mem_alloc
80016bf4 T v3d_mem_free
80016c30 T v3d_mem_lock
80016c64 T v3d_mem_unlock
800211cc D vShaderStr
80000000 A virt_start
8000a340 T writeBitmasked

LdB wrote: Again I don't understand why you are doing all the linker stuff you can just compile the kernel with GCC -fPIC option
PIC stands for "Position Independent Code" and you can put the kernel wherever you want
This is the other option I know people do it. Even I used that in my i386-OS. When I saw this thread I thought I could use a linker script like this.
I know the -fPIC option I just wanted to try this way if it works. Also I was curious which method is better.
LdB wrote: You seem to be trying to make an image file with a physical memory block and a virtual memory block. Even if you could
make such a file how do you propose the VC4 when it picks that file up it would load the virtual memory block given the MMU is off?
What I understand from this linker script is that it will generate single elf file with some of the code having addresses at 0x8000 and the remaining at 2GB. My plan is to load whole kernel file at 0x8000 (boot section and higher half). Execute boot section and enable paging which will map current physical address of higher half code to 2GB, then turn on MMU so all higher code works. Also it doesn't matter what is inside binary file as far as VC4 is concerned. It's job is to find kernel file and load it at requested address, as long as file size is within physical memory limit i.e. 1 GB for PI 3B.

LdB wrote: So he knows the start of his file will start as 0x0 and he needs it become 0xffff000000000000 when he turns the MMU on.
IN SHORT he faked it all don't take the address literally, they become true only if the MMU is setup and turned on correctly.
That is what I am trying to do fake the higher half code addresses until I turn on MMU.
Let there be some light .../\...

bzt
Posts: 373
Joined: Sat Oct 14, 2017 9:57 pm

Re: Virtual memory and linking

Sat Jul 13, 2019 11:39 am

Hi,

You could take a look at my bootloader. It's loaded at 80000, which loads an ELF kernel from the SD card (or over serial) and sets up paging for it, jumping to the entry point in upper-half address. That way your kernel can be an ELF (no objconv needed), purely linked to upper-half, forgetting about all the mixed mapping and jumping trickery stuff. Once your kernel is running, you can change the mapping and remove the loader entirely from the address space if you like (that's what I do when I start my first process). One note, your kernel will be executed on all cores in parallel, so you might want to start your kernel with a "mpidr" check.

Cheers,
bzt

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sat Jul 13, 2019 12:48 pm

Hi bzt,/LdB,


I was trying to get away from this two-stage loading process, but now I think that was the mistake since I am trying tightly couple bootloader with kernel image.

Only problem is how do I debug/test such kernel on qemu? On real hardware bootloader will set up the paging for kernel, what happens in case of qemu?

Thanks for the replies __/\__
Let there be some light .../\...

LdB
Posts: 1143
Joined: Wed Dec 07, 2016 2:29 pm

Re: Virtual memory and linking

Sat Jul 13, 2019 3:32 pm

zeoneo wrote:
Sat Jul 13, 2019 10:37 am
I don't think it's correct. Checkout the kernel map file here (80009000 T __code_start)
I had to sit and work that out and it works because the offset is a 32 bit integer and 0x80000000 is minus 2,147,483,648
That is why it's a subtraction minus a negative becomes a positive :-)
So your section start is 0x8000 then you have . = ALIGN(4096); so it aligns to the 0x1000 boundary which will be 0x9000 assuming you have less than 4K of code in boot (which it appears you must)
Then it does subtracts -2,147,483,648 which adds 0x80000000 so you get your 0x80009000

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sat Jul 13, 2019 4:02 pm

LdB wrote: Then it does subtracts -2,147,483,648 which adds 0x80000000 so you get your 0x80009000
It will be always positive no matter if it is 32 or 64 bits. Label __code_start comes after virt_start hence it is placed at a higher address than virt_start that's what I understand from it.

Any clues on how to test this higher half kernel with 2nd stage loader in qemu?
Let there be some light .../\...

LdB
Posts: 1143
Joined: Wed Dec 07, 2016 2:29 pm

Re: Virtual memory and linking

Sun Jul 14, 2019 3:06 am

zeoneo wrote:
Sat Jul 13, 2019 4:02 pm
It will be always positive no matter if it is 32 or 64 bits. Label __code_start comes after virt_start hence it is placed at a higher address than virt_start that's what I understand from it.
I will agree to disagree it isn't worth worrying about because I would do it that way never, so I really don't care :-)
zeoneo wrote:
Sat Jul 13, 2019 4:02 pm
Any clues on how to test this higher half kernel with 2nd stage loader in qemu?
That is probably a bzt or other question I don't use QEMU at all I do all my debugging on DS5 community.
Either way it's fairly trivial to write because you have MMU table and start code from what I know which is the harder bit.
However I am still trying to get a clear picture of what you are thinking is in and out of the virtualization on the kernel, even things like where do your interrupts go and perhaps where do they get processed? I guess what I am trying to say is in AARCH64 virtualization it is so extreme I really can't take anything for granted and it is hard to offer any advice without more detail..

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sun Jul 14, 2019 5:56 am

LdB wrote: I will agree to disagree it isn't worth worrying about because I would do it that way never, so I really don't care
:) Same here.

I am just posting the answer to my original question on how to use a linker script to generate single kernel binary file (boot section = 0x8000 and higher kernel half at 2GB). I have tested the below solution on qemu as well as real hardware.

Note: Recommended way to achieve higher half kernel is to use second stage loader. I am just using RPI's VC4 to load my kernel at once that way I don't need to write file system reader to load kernel file from file system in early stages.

----------------linker script------------

Code: Select all

ENTRY(_start)

/* Address where boot section will be linked */
load_start = 0x8000;

/*
 *
 *  Address where kernel will be linked
 *  We will initialize paging and map the kernel physical address to
 *  virt_start address. And then jump to kernel code.
*/
virt_start = 0x80000000; /* 2GB */

SECTIONS
{
    /* Starts at LOADER_ADDR. */
    . = 0x8000;
    __start = .;
    __text_boot_start = .;
    .text.boot :
    {
        KEEP(*(.text.boot))
    }
    __text_boot_end = .;
    . = ALIGN(4096);
    __text_boot_end_aligned = .;
    /* Kernel sections are placed after this */
    . = . + virt_start;

	.text : AT(__code_start - virt_start) { 
        __code_start = .;
        *(.text*)
    }
	.rodata : { *(.rodata) }
	.data : { *(.data) }

    __bss_start = .;
    .bss :
    {
        bss = .;
        *(.bss*)
    }
    __bss_end = .;

    . = . + 131072; /* 128KB for svc stack */
    __svc_stack_base = .;

    . = . + 1024; /* 1024 Bytes for irq stack */
    __irq_stack_base = .;

    . = ALIGN(16384); /* align 16KB for page tables */

    __first_lvl_tbl_base = .;
    . = . + 4096; /* First level page table size */
    __first_lvl_tbl_end = .;

    /* 1KB for each 2nd lvl page table = 4 Bytes * 256 */
    /* How many 2nd lvl tables ? */
    __second_lvl_tbl_base = .;
    . = . + 0x00100000; /* 1MB for 2nd level page tables */
    __second_lvl_tbl_end = .;
    __kernel_end = .;

}

-----------------------------------boot.S---------------------------

Code: Select all


// To keep this in the first portion of the binary.
.section ".text.boot"
 
// Make _start global.
.globl _start
.global _get_stack_pointer
.global _exception_table
.global _enable_interrupts

.equ ARM_MODE_USR, 0x10;                                ;@ CPU in USR mode .. Normal User mode
.equ ARM_MODE_FIQ, 0x11;                                ;@ CPU in FIQ mode .. FIQ Processing
.equ ARM_MODE_IRQ, 0x12;                                ;@ CPU in IRQ mode .. IRQ Processing
.equ ARM_MODE_SVC, 0x13;                                ;@ CPU in SVC mode .. Service mode
.equ ARM_MODE_HYP, 0x1A;                                ;@ CPU in HYP mode .. Hypervisor mode  (ARM7/ARM8 only)
.equ ARM_MODE_UND, 0x1B;                                ;@ CPU in UND mode .. Undefined Instructions mode
.equ ARM_MODE_SYS, 0x1F;    

.equ ARM_MODE_MASK, 0x1F;                                ;@ Mask to clear all but CPU mode bits from cpsr register
.equ ARM_I_BIT,        0x80;                                ;@ IRQs disabled when set to 1
.equ ARM_F_BIT,        0x40;

.equ    CPSR_MODE_USER,         0x10
.equ    CPSR_MODE_FIQ,          0x11
.equ    CPSR_MODE_IRQ,          0x12
.equ    CPSR_MODE_SVR,          0x13
.equ    CPSR_MODE_ABORT,        0x17
.equ    CPSR_MODE_UNDEFINED,    0x1B
.equ    CPSR_MODE_SYSTEM,       0x1F

// See ARM section A2.5 (Program status registers)
.equ    CPSR_IRQ_INHIBIT,       0x80
.equ    CPSR_FIQ_INHIBIT,       0x40
.equ    CPSR_THUMB,             0x20

.equ    SCTLR_ENABLE_DATA_CACHE,        0x4
.equ    SCTLR_ENABLE_BRANCH_PREDICTION, 0x800
.equ    SCTLR_ENABLE_INSTRUCTION_CACHE, 0x1000


.balign 4
_start:
    mrc p15, 0, r6,c0,c0,5
    and     r6, r6, #3
    mov r7, #0
    cmp     r6, r7
    beq 2f
    // cpu id > 0, stop
1:  wfe
    b       1b
2:  // cpu id == 0


    mov r12, pc                                            ;@ Hold boot address in high register R12
    mrs r0, CPSR                                        ;@ Fetch the cpsr register
    orr r0, r0, #(ARM_I_BIT | ARM_F_BIT)                ;@ Disable Irq/Fiq
    and r11, r0, #ARM_MODE_MASK                            ;@ Clear all but CPU mode bits in register r11

     cmp r11, #ARM_MODE_HYP                                ;@ Check we are in HYP_MODE                                            
    bne .NotInHypMode                                    ;@ Branch if not equal meaning was not in HYP_MODE  
    bic r0, r0, #ARM_MODE_MASK                            ;@ Clear the CPU mode bits in register r0                            
    orr r0, r0, #ARM_MODE_SVC                            ;@ ARM_MODE_SVC bits onto register    
    msr spsr_cxsf,r0                                    ;@ Hold value in spsr_cxsf
    add lr,pc,#4                                        ;@ Calculate address of .NotInHypMode label

    msr ELR_hyp, lr
    eret

.NotInHypMode:
    // enable unaligned address access
    mrc p15, 0, r0, c1, c0, 0
    orr r0, #1 << 22
    mcr p15, 0, r0, c1, c0, 0

    // Enable VFP ------------------------------------------------------------

    // r1 = Access Control Register
    MRC p15, #0, r1, c1, c0, #2
    // enable full access for p10,11
    ORR r1, r1, #(0xf << 20)
    // ccess Control Register = r1
    MCR p15, #0, r1, c1, c0, #2
    MOV r1, #0
    // flush prefetch buffer because of FMXR below
    MCR p15, #0, r1, c7, c5, #4
    // and CP 10 & 11 were only just enabled
    // Enable VFP itself
    MOV r0,#0x40000000
    // FPEXC = r0
    FMXR FPEXC, r0


    mov r0,#0xD3
    msr cpsr_c,r0
    mov sp, #0x8000 ;@ Temporarily initialize stack pointer to 0x8000 it will grow down to lower memory address
    ;@ After mapping higher half of the kernel we need to reinitialize stack pointer
    ;@ SP is initialized here bcoz I have poor asm skills and I need to call C funtion "initialize_virtual_memory" which might use stack pointer
    ldr r8, =kernel_main

    bl initialize_virtual_memory


    ;@ (PSR_IRQ_MODE|PSR_FIQ_DIS|PSR_IRQ_DIS)
    mov r0,#0xD2
    msr cpsr_c,r0
    ldr sp, =__irq_stack_base

    ;@ (PSR_SVC_MODE|PSR_FIQ_DIS|PSR_IRQ_DIS)
    mov r0,#0xD3
    msr cpsr_c,r0
    ldr sp, =__svc_stack_base

    bx r8 ;@jump kernel

    b _inf_loop

    // halt
_inf_loop:
    b       _inf_loop

.globl BOOT_PUT32
BOOT_PUT32:
    str r1,[r0]
    bx lr

.globl start_mmu
start_mmu:
    mov r2,#0
    mcr p15,0,r2,c8,c7,0 ;@ invalidate tlb
    mcr p15,0,r2,c7,c10,4 ;@ DSB ??

     ;@ Domain 0 Manager access. No permissions checked for access
    MRC p15, 0, r2, c3, c0, 0 ;@ Read DACR into Rt
    ORR r2, #3
    MCR p15, 0, r2, c3, c0, 0 ;@ Write Rt to DACR

    MCR p15,0,r0,c2,c0,0 ;@ Write Rt to TTBR0
    MCR p15, 0, r0, c2, c0, 1 ;@ Write Rt to TTBR1

    MRC p15, 0, r2, c1, c0, 0 ;@ Read SCTLR into Rt
    orr r2,r2,r1
    MCR p15, 0, r2, c1, c0, 0 ;@ Write Rt to SCTLR.

    bx lr


C file where I initialize paging.
-------------------------------------------

Code: Select all



void __attribute__((section (".text.boot"))) initialize_virtual_memory(void)
{

    uint32_t MMUTABLEBASE = (uint32_t)&__text_boot_end_aligned + (uint32_t)&__first_lvl_tbl_base - (uint32_t)&__code_start;

    uint32_t SECOND_TBL_BASE = (uint32_t)&__text_boot_end_aligned + (uint32_t)&__second_lvl_tbl_base - (uint32_t)&__code_start;
    
    // Identity Map boot.text section 
    // TODO: Convert this to loop instead of hardcoding addresses.
    mmu_page(0x8000, 0x8000, 0x0000, MMUTABLEBASE, SECOND_TBL_BASE);

    // Map Higher half kernel
    uint32_t ra = (uint32_t)&__text_boot_end_aligned;
    uint32_t higher_half_kernel_end = (uint32_t)&__text_boot_end_aligned + (uint32_t)&__second_lvl_tbl_end - (uint32_t)&__code_start;
    uint32_t virt_addr = 0x80000000;
    while(1) {
        mmu_section(virt_addr, ra, 0x0000, MMUTABLEBASE);
        ra += 0x00100000; // 1MB
        virt_addr += 0x00100000;
        if(ra > (higher_half_kernel_end + 0x00100000)) {
            break;
        }
    }

    
    //peripherals
    mmu_section(0x80000000 + 0x3f000000, 0x3f000000, 0x0000, MMUTABLEBASE); //NOT CACHED!
    mmu_section(0x80000000 + 0x3f200000, 0x3f200000, 0x0000, MMUTABLEBASE); //NOT CACHED!

    start_mmu(MMUTABLEBASE, 0x00000005);
}

uint32_t  __attribute__((section (".text.boot"))) mmu_section(uint32_t vadd, uint32_t padd, uint32_t flags, uint32_t mmu_base)
{
    uint32_t table1EntryOffset;
    uint32_t table1EntryAddress;
    uint32_t tableEntry;

    table1EntryOffset = (vadd >> 20) << 2; // get only most significant 12 bits
    //and multiply it by 4 as each entry is 4 Bytes 32bits

    // MMU table base should be at 16KB granularity, Least signficant 12 bits will be always 0. hence do OR with that
    table1EntryAddress = mmu_base | table1EntryOffset;

    // 31: 20  12 bits are physical 12 ms bits from physical address
    tableEntry = (padd & 0xFFF00000);

    // entry[1:0] = 0b10 for section entry
    tableEntry = tableEntry | 2;

    // Access permissions should be 11 for full access entry [11:10] = 0b11
    tableEntry = tableEntry | 0xC00;

    // Add flags
    tableEntry = tableEntry | flags;

    //hexstrings(rb); hexstring(rc);
    // printf("\n entryAddr: 0x%x, entry value:0x%x \n", table1EntryAddress, tableEntry);
    BOOT_PUT32(table1EntryAddress, tableEntry);
    return (0);
}

uint32_t __attribute__((section (".text.boot"))) mmu_page ( uint32_t vadd, uint32_t padd, uint32_t flags, uint32_t first_lvl_base, uint32_t second_lvl_base)
{
    uint32_t ra;
    uint32_t rb;
    uint32_t rc;

    ra= vadd>>20;
    rb= first_lvl_base|(ra<<2);
    rc= (second_lvl_base & 0xFFFFFC00) |1;
    BOOT_PUT32(rb,rc); //first level descriptor
    ra=(vadd>>12)&0xFF;
    rb=(second_lvl_base&0xFFFFFC00)|(ra<<2);
    rc=(padd&0xFFFFF000)|(0xFF0)|flags|2;
    BOOT_PUT32(rb,rc); //second level descriptor
    return(0);
}

Thanks,
zeo
Let there be some light .../\...

zeoneo
Posts: 52
Joined: Sun Sep 30, 2018 6:54 am

Re: Virtual memory and linking

Sun Jul 14, 2019 6:13 am

LdB wrote: However I am still trying to get a clear picture of what you are thinking is in and out of the virtualization on the kernel, even things like where do your interrupts go and perhaps where do they get processed?

I am planning to run all the EL1 stuff from 2GB on wards. Userspace code will run in virtual address space 0x0 -> 2GB.

ttbr0_el0 -> will contain a mapping for userspace code.
ttbr1_el0 -> mapping for kernel code.

I will reinitialize VBAR (Vector base address register in kernel virtual address space). All the syscalls and interrupts processing will happen in el1.
Let there be some light .../\...

Return to “Bare metal, Assembly language”