Yet Another Bare Metal Tutorial for the RPi3

[Ron02]

Dear all,

I reviewed the Posts in this Forum, but I didn't find anything About Debugging the Pi from a IDE. The Arduino has a EDBG interface integrated so that a remote debugging from the host PC is feasible. Is there some opportunity for doing this with the Raspberry Pi as well or do I have to use the SD card even for bare metal programming?

Thanks in advance

Ron

[bzt]

Hi Ron,

As far as I see, you have the several options:
1. you can use raspbootin (I've rewritten it for 64 bit) to avoid SD card usage and boot your kernel over serial line
2. for debugging, you can compile in my mini-debugger and use that over serial line with any terminal emulator (even with rasbootcom)
3. most complicated, but most promosing for fully featured IDE itegration is to compile a gdb remote stub into you kernel. In theory gdbserver has a patch for AArch64, but honestly I haven't tried that. There's also a (not very helpful) description on ARM info center on how to use JTAG in virtual ethernet/tty mode with gdb.
4. if you are fine with a virtual environment, qemu has a built-in disassembler (-d int,in_asm) and built-in gdb server (-S -s) which you can use without modifying your kernel just out-of-the-box.

Cheers,
bzt

[pxlnpx]

3. most complicated, but most promosing for fully featured IDE itegration is to compile a gdb remote stub into you kernel. In theory gdbserver has a patch for AArch64, but honestly I haven't tried that. There's also a (not very helpful) description on ARM info center on how to use JTAG in virtual ethernet/tty mode with gdb.

Being new to binutils and related tools, a quick bare-metal debugging-related question: how to modify the ‘link.ld’ script in bzt’s tutorials in order to contiguously include all debugging info (i.e. dwarf .debug_* sections) and how would e.g. a running program later discover its own debug info memory starting address and its size?

[bzt]

Being new to binutils and related tools, a quick bare-metal debugging-related question: how to modify the ‘link.ld’ script in bzt’s tutorials in order to contiguously include all debugging info (i.e. dwarf .debug_* sections) and how would e.g. a running program later discover its own debug info memory starting address and its size?

Add "-g" to CFLAGS in Makefile. Then load "kernel8.elf" into gdb while running "kernel8.img" as usual. I think that's all (the elf executable already contains the starting address and size for each segment, and "-g" adds dwarf sections and symbol translations to it). You can add a new segment in link.ld for debug info, but shouldn't be needed. Normally they are just appended to the text segment. If memory serves gdb may need some gnu specific sections though (like gnu hash I think), so you should remove (.gnu*) from the DISCARD rule just to be on the safe side.

Cheers,
bzt

[pxlnpx]

... You can add a new segment in link.ld for debug info, but shouldn't be needed. Normally they are just appended to the text segment. ...

Thanks for this hint; I just added

Code: Select all

    .debug_info 0 : {
        __debug_info_start = .;
        *(.debug_info)
        __debug_info_end = .;
    }
__debug_info_size = SIZEOF(.debug_info);

for all .debug_* sections and 'objdump' shows the linker now does what one expects. Unfortunately

Code: Select all

    objcopy -O binary kernel8.elf kernel8.img

ruins this success by arbitrarily removing all debugging info (one does not have to specify '-R' or '-g' for this to happen, even passing '-debugging' to 'objcopy' in the hope the debug sections simply get flushed into the binary doesn't change anything here.)

Is there any alternative tool to turn ELF files into binary, without loosing any sections?

[bzt]

...ruins this success by arbitrarily removing all debugging info

This shouldn't be a problem. You run the .img, true, but you load the .elf (with all the sections) into gdb. Therefore gdb will be able to read the debug information and the symbols even though the running .img doesn't have them.
But if you want to keep the debug info in the .img regardless, simply put them in the text section after the rodata.

Cheers,
bzt

[pxlnpx]

...But if you want to keep the debug info in the .img regardless, simply put them in the text section after the rodata.

Thanks for this hint, it looks promising.
Though, by this simple copy

Code: Select all

.text : {
    *(.debug_info)
}

the LMA and VMA addresses look different, and the dwarf info unfortunately gets "corrupted" too (is different if embedded into a .text section). Most probably the corrupted debug-info from the .text section gets dumped into the baremetal image by "objdump -O binary ..." too (haven't tested though).

[pxlnpx]

In the process of studying the arm processor by following bzt's tutorials an interesting question emerged: can raspberrypi be deployed in a big-endian mode? Just as a toy exercise here is bzt's tutorial "03_uart1" slightly modified to support big-endian. The "start.S" file:

Code: Select all

.section ".text.boot"

.global _start

_start:
    // read cpu id, stop slave cores
    mrs     x1, mpidr_el1
    and     x1, x1, #3
    // cpu id > 0, stop
    cbnz    x1, 1f

    // set stack before our code
    adr     x1, _start
    msr     sp_el1, x1

    // enable AArch64 in EL1
    mov     x2, #(1 << 31)      // AArch64
    orr     x2, x2, #(1 << 1)   // SWIO hardwired on Pi3
    msr     hcr_el2, x2
    mrs     x2, hcr_el2

    // Setup SCTLR access
    mov     x2, #0x0800
#if (defined __BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
    movk    x2, #0x33d0, lsl #16
#else
    movk    x2, #0x30d0, lsl #16
#endif
    msr     sctlr_el1, x2

    // change execution level to EL1
    mov     x2, #0x3c4  //  PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h
    msr     spsr_el2, x2
    adr     x2, 5f
    msr     elr_el2, x2
    eret

5:  mov     sp, x1

    // clear bss
    ldr     x1, =__bss_start
    ldr     w2, =__bss_size
3:  cbz     w2, 4f
    str     xzr, [x1], #8
    sub     w2, w2, #1
    cbnz    w2, 3b

    // jump to C code, should not return
4:  bl      main
    // for failsafe, halt this core too

1:  wfe
    b       1b

(From the "_start:" label until the "eret" instruction we possibly execute big-endian code on a little-endian cpu.) The mailbox-interface apparently is only little-endian by design, so "uart.c" had to be refactored a little:

Code: Select all

#include "gpio.h"

/* Auxilary mini UART registers */
#define AUX_ENABLE      ((volatile unsigned int*)(MMIO_BASE+0x00215004))
#define AUX_MU_IO       ((volatile unsigned int*)(MMIO_BASE+0x00215040))
#define AUX_MU_IER      ((volatile unsigned int*)(MMIO_BASE+0x00215044))
#define AUX_MU_IIR      ((volatile unsigned int*)(MMIO_BASE+0x00215048))
#define AUX_MU_LCR      ((volatile unsigned int*)(MMIO_BASE+0x0021504C))
#define AUX_MU_MCR      ((volatile unsigned int*)(MMIO_BASE+0x00215050))
#define AUX_MU_LSR      ((volatile unsigned int*)(MMIO_BASE+0x00215054))
#define AUX_MU_MSR      ((volatile unsigned int*)(MMIO_BASE+0x00215058))
#define AUX_MU_SCRATCH  ((volatile unsigned int*)(MMIO_BASE+0x0021505C))
#define AUX_MU_CNTL     ((volatile unsigned int*)(MMIO_BASE+0x00215060))
#define AUX_MU_STAT     ((volatile unsigned int*)(MMIO_BASE+0x00215064))
#define AUX_MU_BAUD     ((volatile unsigned int*)(MMIO_BASE+0x00215068))

unsigned int get32le (volatile unsigned int *p)
{
#if (defined __BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
    return __builtin_bswap32 (*p);
#else
    return (*p);
#endif
}

void put32le (volatile unsigned int *p, unsigned int i)
{
#if (defined __BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
    (*p) = __builtin_bswap32 (i);
#else
    (*p) = i;
#endif
}

/**
 * Set baud rate and characteristics (115200 8N1) and map to GPIO
 */
void uart_init (void)
{
    register unsigned int r;

    /* initialize UART */
    put32le (AUX_ENABLE, get32le (AUX_ENABLE) | 1);  // enable UART1, AUX mini uart
    put32le (AUX_MU_CNTL, 0);
    put32le (AUX_MU_LCR, 3);  // 8 bits
    put32le (AUX_MU_MCR, 0);
    put32le (AUX_MU_IER, 0);
    put32le (AUX_MU_IIR, 0xC6);  // disable interrupts
    put32le (AUX_MU_BAUD, 270);  // 115200 baud

    /* map UART1 to GPIO pins */
    r  = get32le (GPFSEL1);
    r &= ~((7<<12) | (7<<15));  // gpio14, gpio15
    r |=  ((2<<12) | (2<<15));  // alt5
    put32le (GPFSEL1, r);

    put32le (GPPUD, 0);  // enable pins 14 and 15
    r = 150; while (r--) { asm volatile ("nop"); }
    put32le (GPPUDCLK0, (1<<14) | (1<<15));
    r = 150; while (r--) { asm volatile ("nop"); }
    put32le (GPPUDCLK0, 0);  // flush GPIO setup
    put32le (AUX_MU_CNTL, 3);  // enable Tx, Rx
}

/**
 * Send a character
 */
void uart_send (unsigned int c)
{
    /* wait until we can send */
    do { asm volatile ("nop"); } while (!(get32le (AUX_MU_LSR) & 0x20));
    /* write the character to the buffer */
    put32le (AUX_MU_IO, c);
}

/**
 * Receive a character
 */
char uart_getc (void)
{
    char r;
    /* wait until something is in the buffer */
    do { asm volatile ("nop"); } while (!(get32le (AUX_MU_LSR) & 0x01));
    /* read it and return */
    r = (char)(get32le (AUX_MU_IO));
    /* convert carrige return to newline */
    return r == '\r' ? '\n' : r;
}

/**
 * Display a string
 */
void uart_puts (char *s)
{
    while (*s) {
        /* convert newline to carrige return + newline */
        if (*s == '\n')
            uart_send ('\r');
        uart_send (*s++);
    }
}

Compiled using

Code: Select all

    gcc -ffreestanding -nostdinc -nostdlib -nostartfiles -mbig-endian -o kernel8.elf -T link.ld start.S main.c uart.c
    objcopy -O binary kernel8.elf kernel8.img

There are still few open questions:

• could start.S be reduced in the sense of fewer assembler instructions?
• what is the purpose of "msr sp_el1, x1", or why do we need "mov sp, x1"?
• is it correct that cores 1, 2, 3 stay in EL2?
• how power-consuming is "1: b 1b"?
• the mailbox-interface is certainly not thread-safe? (Just for the case we later invent main0(), ..., main3() for cores 0, ..., 3 to enter, and just want to access mailbox from any of them.)

Sorry for this big post.

[bzt]

the dwarf info unfortunately gets "corrupted"

I'm sorry, I can't give you cut out instructions here. The best advice I can give you is to try different configurations until the addresses in dwarf info became correct. Having debug info in the elf only and not in the img shouldn't be a problem, so including debug section into text segment is just optional.

In the process of studying the arm processor by following bzt's tutorials an interesting question emerged: can raspberrypi be deployed in a big-endian mode?

Yes, and you have already done that It worth mentioning that if you implement virtual memory, you must set big-endian enable bits in the paging tables too.

There are still few open questions:

• could start.S be reduced in the sense of fewer assembler instructions?

Probably. My goal was to create easily distinguishable blocks for education, and not optimalization. Although _start is quite small, and only executed once during boot, so optimization doesn't worth it imho.

[*] what is the purpose of "msr sp_el1, x1", or why do we need "mov sp, x1"?

Probably. The first one sets the stack for exception handlers (while running in EL2), the second one sets the current sp (running in EL1). There's a good chance you never start your kernel at EL1, therefore the eret is always executed, and sp is always loaded from sp_el1.

[*] is it correct that cores 1, 2, 3 stay in EL2?

Yes. Simplicity was my goal. For a full-blown implementation, take a look at my bootloader's boot.S. It sets up all cores (EL1, virtual mappings, etc.), loads an ELF from an initrd, maps it in higher half (-2M) and starts executing it on all cores. Except for the stack, all cores are intialized equally.

[*] how power-consuming is "1: b 1b"?

Very much. It's generating 100% CPU usage. Use "1: wfe; b 1b" instead. The Wait For Event instruction puts the cpu core in a low energy consuption mode until it receives an interrupt (or some other similar event).

[*] the mailbox-interface is certainly not thread-safe? (Just for the case we later invent main0(), ..., main3() for cores 0, ..., 3 to enter, and just want to access mailbox from any of them.)

No, you should implement an exclusive access mechanism for it. Simpliest is a spinlock, so that only one CPU can write the mailbox MMIO address at any given time. By the way, the same stands for all MMIO addresses. It's not healthy either if more CPUs are trying to write the same UART registers concurrently for example.

Sorry for this big post.

Don't you worry

Cheers,
bzt

[bzt]

the LMA and VMA addresses look different, and the dwarf info unfortunately gets "corrupted" too (is different if embedded into a .text section). Most probably the corrupted debug-info from the .text section gets dumped into the baremetal image by "objdump -O binary ..." too (haven't tested though).

Hi,

I've tested this for you with my bootloader. These are the steps I've done:
1. I've added "-g" to aarch64-elf-gcc in the Makefile (to generate debug info)
2. I haven't changed anything in the linker script
3. The bootboot.elf's size increased significantly, but bootboot.img remained the same (text segment unchanged)

Now in one terminal, I've started qemu like this (-s stops guest execution, -S starts the built-in gdb-server):

Code: Select all

$ qemu-system-aarch64 -s -S -M raspi3 -kernel bootboot.img

In another terminal, I've started the cross-platform gdb that my distro ships:

Code: Select all

$ aarch64-linux-gnu-gdb

Then in the gdb prompt, I've typed

Code: Select all

(gdb) set architecture aarch64
The target architecture is assumed to be aarch64

To set AArch64 architecture.

Code: Select all

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x0000000000000000 in ??()

To connect the gdb to the gdb-server in qemu.

Code: Select all

(gdb) symbol-file bootboot.elf
Reading symbols from bootboot.elf...done.

To load the symbols and debugging information from the elf file.

Code: Select all

(gdb) display/i $pc
1: x/i $pc
=> 0x0: ldr     x0, 0x18

I've used this because I like to see what's the machine code doing

Code: Select all

(gdb) break bootboot_main
Breakpoint 1 at 0x840c0: file bootboot.c, line 1110.

As a test, I've set up a breakpoint at one of the C functions. As you can see, the address was read from the elf symbols correctly, and the debug info provided the source file and line also correctly.

Code: Select all

(gdb) c
Continuing.

Thread 1 hit Breakpoint 1, bootboot_main (hcl=2147483650) at bootboot.c:1110
1110    {
1: x/i $pc
=> 0x840c0 <bootboot_main>:     sub      sp, sp, #0xa90

Finally with "c" (stands for continue), I have started the virtual machine. The execution then stopped at my function as expected, showing the source line (that's just a "{" block opening in this case) and the first instruction.

For a better view, type "layout split". That will show you several lines from the source file with the disassembed instructions in parallel above the gdb prompt, like this (note this is just an example I found on the internet, it's actually x86):
.

Hope this helps.

Cheers,
bzt

[NeoFahrenheit]

Hi.

I've recently created a topic asking about how to create a operating system for the Pi 3. A user recommended me to look at this sub-forum and I have found your topic. I'm looking for advice to see if this series of tutorials is what I need. First, if you don't mind, I would like to say what my objective is. Forgive my english, please.

At my University I had a project where I needed to work in a "emulated processor in software". We needed to create a program and a kernel which our programs calls functions to. It was very interesting. Now I'm thinking to move it to the next level.

So, I want to create a multi-threaded O.S. for my Raspberry Pi 3 B+. I think I need to know about the processor architecture first. Learn to know how the program counter works, the stack pointer, the clock, the interrupts, etc. Then I would start simple and make a command-line interface only. Later, I would love to try to make a GUI.

I've pickup up Operating Systems Internals and Design Principles, by Stallings, Operating Systems Design and Implementation, by Tanenbaum Woodhull and Data and Computer Communications, by Stallings in the library. The first two is suppose to help me understand how O.S. works and the third to "give my O.S. internet capabilities" skills.

Feel free if you want to say "that book is crap! Pick another!" or anything.
Well, my question is: Do you think this tutorial is a good place to start? I already know C and C++.

Thank you very much!

[gpk]

Why does this code use 0x3f215000 as the AUX peripherals map when the BCM2835 documentation says its 0x7e215000? I noticed that Linux uses the latter while this code (and others) use the former.

[LdB]

You have a shared memory bus, the 0x7e215000 address is from the GPU the VC4 which is the main processor the 0x3f215000 is from the ARM processor which is a co-processor to the VC4. They represent the same point in memory from two different processors. You need to remember the VC4 does all the disk image loading and actually sets up the code the ARM processor reset will be released and run into.

[bzt]

Hi,

Feel free if you want to say "that book is crap! Pick another!" or anything.
Well, my question is: Do you think this tutorial is a good place to start? I already know C and C++.

No, those books are great sources. I'd suggest to get Tanenbaum's and Bos' book too, it's called Modern Operating Systems 4th edition. It is more general than the Tanenbaum book you mentioned, and instead of listing the Minix C source it has a very detailed analysis on how Minix, Linux and Windows kernels work. That is a very valuable information specially for beginners.

My tutorial won't teach you how to create an operating system and it doesn't cover theory at all. Instead it's intended to help you out with specific parts, like how to read a file from disk without all the VFS stuff. Or how to get an initrd loaded on the RPi. There are more tutorials like mine, I've linked them in the main README.md check them out. There's one tutorial in particular which focuses on how Linux does things on the RPi, and how a small hobby OS can do things. Unfortunatelly unfinished, but raspberrypi-os is still extremely useful. Also linked on my main README.md.

Welcome to the club and Good luck,
bzt

[NeoFahrenheit]

No, those books are great sources. I'd suggest to get Tanenbaum's and Bos' book too, it's called Modern Operating Systems 4th edition. It is more general than the Tanenbaum book you mentioned, and instead of listing the Minix C source it has a very detailed analysis on how Minix, Linux and Windows kernels work. That is a very valuable information specially for beginners.

My tutorial won't teach you how to create an operating system and it doesn't cover theory at all. Instead it's intended to help you out with specific parts, like how to read a file from disk without all the VFS stuff. Or how to get an initrd loaded on the RPi. There are more tutorials like mine, I've linked them in the main README.md check them out. There's one tutorial in particular which focuses on how Linux does things on the RPi, and how a small hobby OS can do things. Unfortunatelly unfinished, but raspberrypi-os is still extremely useful. Also linked on my main README.md.

Welcome to the club and Good luck,
bzt

Thank you. My biggest fear is how to make the Pi "see and execute" my code, but I already saw that raspberrypi-os has that info.
Now is the study time.

I guess the 4th Edition of the Tanenbaum's book is better, but my library doens't have it. I'll look elsewhere where I can find it.

[bzt]

Thank you. My biggest fear is how to make the Pi "see and execute" my code, but I already saw that raspberrypi-os has that info.
Now is the study time.

Not sure what you mean by "see and execute", you simply create an elf, copy the text segment out into kernel.img and save that on the SD card. That's all

I guess the 4th Edition of the Tanenbaum's book is better, but my library doens't have it. I'll look elsewhere where I can find it.

Not particularly better, I'd say different. It does not focus that much on Minix, instead it has more comparitions and more theory. It's on github btw, also here.

Cheers,
bzt

[Gavinmc42]

Thanks for the book names, I work 50m from a Uni library
QA76.76 is good plus to start looking, I will have to go back for some other books I spotted .
Got the 3rd edition and the Minix Design one too.

I made a Minix 68000 PC from a kit, the AT1616 years ago but had no idea how the software worked, time to fix that.
I have been using Ultibo for baremetal, but have no idea how OS's work.
Most of the old stuff goes right down to basics, not much in the new ways to do things.
Ultibo is sort of like having a kit of OS parts without all the instructions.

The differences between the 3rd and 4th is interesting too.
Pi's sit in weird timeframe. the hardware is oldish, so old stuff like Symbian is relatable.
Android chips are SoC so not mainframe CPU and so the hardware is similar too.

Yet Pi's can run modern Linux etc.
There are hardly any books on OpenVG, yet that is very usable even in baremetal.
And it is much more fun than playing with pixels or brain damaging your head trying to think in 3D and OpenGLES.

[gpk]

Another book that is very good is "Operating System Design, The Xinu Approach" by Douglas Comer. It cover the implementation of a small embedded OS called Xinu. Xinu also runs on Raspberry Pi, there is a port here: https://github.com/LdB-ECM/Xinu (note I had to do some changes to get it to build, there is a PR there).

These bare metal tutorials are great, by the way. I like the fact that it's not too spelled out, and mostly contained in the code itself, which forces one to go and look at the reference manuals to really understand things.

One thing I haven't figured out is what the "command line" property is for on the VC mailbox property channel. For me I just get back an empty string. Is there any better documentation for the mailbox? Am I right in thinking this protocol is implemented in software inside the start.elf for the VC?

[StevoD]

Another book that is very good is "Operating System Design, The Xinu Approach" by Douglas Comer. It cover the implementation of a small embedded OS called Xinu. Xinu also runs on Raspberry Pi, there is a port here: https://github.com/LdB-ECM/Xinu (note I had to do some changes to get it to build, there is a PR there).

The Xinu sources for Pi 1

https://github.com/xinu-os/xinu

or the offical Pi2/3 port

https://github.com/rlatinovich/xinu

might be a better reference to start from

[LdB]

Personally I would ignore Xinu it is pretty much dead

I have 3 pushes been sitting there for 6 months and it has not been updated in a year .. you can see on the issues last 3 are mine
https://github.com/xinu-os/xinu/issues
The most significant is the lan driver on the Pi3B+ which all ethernet use is dead without the lan78xx driver.

There main repo has a pile of quicky things even down to the build system which needed python and shell commands and there are large blocks of code simply commented out into limbo at the moment and no active maintenance.

Rades site similarly has not been updated for ages and his multicore on the Pi2/2 is not fully running yet.

There is no AARCH64 version of Xinu and I am not going to port because there is little interest.

For education O/S you are better of with Minix3 and XV6/SV6 both of which have full multicore support and AARCH32 and AARCH64 versions.
XV6 /SV6 has the most complete set of drivers (wifi is the only significant missing) but the Minix3 system is more interesting for multicore.
There are many active developments of both on github.

[StevoD]

Personally I would ignore Xinu it is pretty much dead

Looks plenty active to me, recent commits and activity. It is an academic project so you wouldn't expect a weekly commit would you?

I have 3 pushes been sitting there for 6 months and it has not been updated in a year .. you can see on the issues last 3 are mine
https://github.com/xinu-os/xinu/issues
The most significant is the lan driver on the Pi3B+ which all ethernet use is dead without the lan78xx driver.

I don't think you understand how git works, did you really expect them to respond to your issues when you don't supply any code or a pull request to back it up, it would end up just wasting their time.

[LdB]

Leaving aside it is all my fault that I can't use git and help them and it is an academic project so we should not expect weekly updates. Ultimately I have running code the site doesn't, it is no issue to me and not my problem to fix.

The last site commit was 7 months ago ( brylow committed on Oct 27, 2018 ) and the code has more than a few issues currently. It won't work on a Pi3B+ or a Pi Zero at all so it's a bit hard to recommend it for someone who comes on the forum looking for code.

Development projects on the internet ebb and flow, we understand that and currently there are better options.

[StevoD]

Ultimately I have running code the site doesn't

That's a bit debatable really, its hard to follow what you've done because you didn't create a fork and seem to replace every file when you do a commit. Your most recent addition, five months ago, was some stuff to do with critical sections which looks like it was written by a total beginner.

Look at this from your platforms/arm-rpi/CriticalSections.c

Code: Select all

static bool inCrit = true;
static uint32_t intmask = 0;

extern uint32_t disable(void);        // In assembler file system/arch/arm/intutils.S
extern void restore(uint32_t mask);   // In assembler file system/arch/arm/intutils.S

void ENTER_KERNEL_CRITICAL_SECTION (void)
{
	if (inCrit == false) {
		kprintf("Aborting .. ENTER_KERNEL_CRITICAL_SECTION called twice without leaving \n");
		while (1) {}
	}
	intmask = disable();
	inCrit = true;
}

void EXIT_KERNEL_CRITICAL_SECTION (void)
{
	if (inCrit == false) {
		kprintf("Aborting .. EXIT_KERNEL_CRITICAL_SECTION called twice without entering \n");
		while (1) {}
	}
	restore(intmask);
}

Why the hell would you use a couple of global variables for that, its miles worse even than the big kernel lock that Linux was hammered for years ago. And worse you didn't even realize that the intmask returned from disable allows you to do recursive calls if you store it on the stack.

Development projects on the internet ebb and flow, we understand that and currently there are better options.

There might be better options but I don't think your offering is one of them.

[LdB]

Well happy to be the total beginner that got everything working ... where is your code pro?

I stopped working on it 5 months ago because it became obvious the main project had gone into hibernation you know about the time I posted issues

I could tell you what that was all about and where the code comes from but it's actually more amusing leaving that stand, and lets be honest you aren't really interested unless it is attacking me.

Now I really don't think the forum needs this sort of interaction so our interactions are done.

[StevoD]

Well happy to be the total beginner that got everything working ... where is your code pro?

There's a big difference between what looks like it works in a basic test and what is usable in real code.

I use the xinu code from github with my own additions, that's the point.

I could tell you what that was all about and where the code comes from but it's actually more amusing leaving that stand

You wrote what you wrote don't make excuses, if you can do better then you should do so, simple as that.

Now I really don't think the forum needs this sort of interaction so our interactions are done.

I'm happy to end any discussion with you, but make sure the advice you freely hand out is accurate, or you might get another reminder.

Here's a hint too, if you're a Windows user check out SourceTree it makes git easier for you.