smccain
Posts: 19
Joined: Tue Dec 04, 2012 6:51 pm

Bootloader Questions

Thu Dec 06, 2012 6:17 pm

I see a lot of posts on here about bootloaders and I've visited the github that dwelch has and read the bootloader examples. I understand the process complete as I'm not new to computer engineering, but what I don't understand or get is how user programs are stored permanantely.

From what I can tell all the bootloaders listen on rs232 (typical for mcu solutions) and they implement various protocols but what it all revolves around is basically the stm instruction which allows you to store data in memory. After the bootloader is done reading the user program it usually performs a reset to the user program space.

Here is where I start to question the process. Since this the memory is volatile the contents will be erased on the next power cycle. That means the user program your bootloader so painstakingly copied will be gone at the next power cycle. I fail to see the usefulness of that. That would be like everytime I needed to boot my PC I had to connect it to the internet and download windows before I can start working.

So, my question is how can this be done so it's more of a permanent solution?

A couple of thoughts (and I have no idea if this is possible or not):

1. Have the bootloader program also copy the user program to the sd card. When the bootloader starts and can wait for any activity on rs232, if no activity it can check for existence of the file and load from sdcard to memory and then switch to user code.

2. Have the user program tacked onto kernel.img at a certain offset (controlled by linker scripts) and have the bootloader read that into memory.

There are some disadvantage to both approaches, but I think the most reusable one which would require the least amount of interference with the device would be the first one. I can imagine with the first option you could have the device sitting in the field for years without needing to change anything. The user code can be changed at will via rs232 and it will always defer to the previous user image.

Any thoughts?

Vassius
Posts: 25
Joined: Sun Jun 03, 2012 7:56 pm

Re: Bootloader Questions

Thu Dec 06, 2012 7:19 pm

The bootloader obviously isn't intended for persistent programs. It's intended for use during the development phase, so that you won't have to copy your program/kernel to the SD card every time you have an update to evaluate.

If you want it persistent, just replace the bootloader with your program on the SD card.

dwelch67
Posts: 961
Joined: Sat May 26, 2012 5:32 pm

Re: Bootloader Questions

Thu Dec 06, 2012 7:25 pm

The non-volatile storage in this system is the sd card. If you want to have a permanent solution then simply place your application (kernel.img) on the sd card. or have one card with a bootloader for development and another sd card for each application that you want to just come up and run. (or a raspi plus sd card for every application you want to run, whatever combination of the above).

The alternative would be to make a more complicated bootloader that somehow decides to wait for commands or boot the last saved program (perhaps a timeout or a shorted pin or something). And if in bootloader mode then have the bootloader save the program to the sd card which is the non-volatile memory in a typical raspi system. One/some folks may have already done this. A usb based solution would be cool and beyond that an ethernet (which in this case is through usb) would also be nice.

My bootloaders are very very simple and brute force, the goal to save on the sd card dance, mechanical damage, wear and tear. It still wears on the usb cable and a connector (some folks have cut the usb cable and put a switch on the power line). Also to save time if/when you have to do this dozens to hundreds of times for an application.

-on sandbox-
My goal is to hopefully gently help folks get started and the let them go off and create something. My tools are not meant to be applications (maybe the bootloader but not really even that). My belief is that folks starting with bare metal are met with a lot of fear and/or failure, it is very very hard without experience or mentoring to debug the many problems getting to a point where you can see signs of life from your code (an led or uart output, etc). Even with my level of experience I have a hard time with new to me platforms from time to time (raspi mini uart for example). If I can get a higher percentage of folks over that threshold, that gives a greater bare metal user base. For as fun as it might be to program in the next greatest scripting language on the next greatest operating system, there still needs to be someone that can build the compiler internals. Can make the bootloader for the systems. Can create the software to test the newly designed system to make sure it can run an operating system, etc. If I can encourage or create even one person out there that can work comfortably at this level then I have at least replaced myself when I am no longer around.
-off soapbox-

Looks like someone has already answered before me. And that is the very concise short answer. My bootloader is very simple and brute force meant for one purpose. Once you are ready to have your program boot instead of be loaded through the bootloader then simply copy it to an sd card and boot off of that sd card. I have no plans to add any file system features (nv storage) to any of my bootloaders. I encourage anyone who wants more features to create their own utility. The reailty is that I created the serial bootloader for folks that might not be able or willing to pay for a jtag device ($50+ vs $20- for a serial interface) and/or may not be able to solder the required wire (that requirement is on its way out from what the raspi folks have said). Professionally and personally I use jtag with cheap wigglers where possible.

smccain
Posts: 19
Joined: Tue Dec 04, 2012 6:51 pm

Re: Bootloader Questions

Thu Dec 06, 2012 7:49 pm

Thanks for all the replies. I appreciation the pedantry but it's lost on me. I'm not "afraid" of baremetal programming, I just want to understand the ARM platform. I'm not new to MCU's having sharpened my teeth on 8051 and PIC's back in the late 90s and early aughts and moved to AVR's from there.

Also, worry not, I'm not using your bootloader. I wrote my own. A bootloader isn't something magical. It either loads data from disk, rom (bios), or some other external source - in this case rs232. Hence the name bootloader. You pull yourself up by your bootstraps with the bootstrap being the small code you write for that.

As for making the bootloader more complex, I don't think it will be much more complex. Heck Arduino has a bootloader that does exactly what I said (sans the SD card access) and those are trivial to write (I know, I've written two). How hard is it to set a timeout, via an interrupt? You sit and spin on the com port until that timeout interrupt comes and then you just branch to the sd loading code. Pretty simple and trivial if you ask me. The only complexity would be the FAT16 code, but there are scads of libraries out there for that.

I built my own cross compiled GCC (Code Sourcery is hiding their free package under layers of links complete with an email registration so I decided to roll my own - I didn't use the scrips on dwelche's site but snarfed the build scripts from an old Code Sourcery source package I had laying around) and am working on my newlib port, so the bootloader is another milestone towards my goal.

My goal is to make an environment that is as easy to use as Arduino is now. Currently there is still a barrier to entry with the raspberry pi. You have to know some basic linux commmands (or learn them) in order to be productive with it. With a baremetal approach I eliminate that layer and am hoping to create a processing-ish environment to code in and having a user program that "sticks" is key to that. I would like to see people with very little knowledge of programming and Linux operating systems take advantage of this super low cost, extremely powerful platform.

Thanks for all the replies.

dwelch67
Posts: 961
Joined: Sat May 26, 2012 5:32 pm

Re: Bootloader Questions

Thu Dec 06, 2012 8:12 pm

I wasnt talking about you specifically i was talking about the purpose of my code, that is all, folks that can already do this might only gain a little from my code where myself or someone else has hacked through the poor documentation, or if you have never used arm before a little kick start moving from one platform to another.

There is more complexity in going from a dumb hundred or so lines of xmodem serial receiver thing to adding sdcard access and file system support, that is the complexity I am talking about. Taking it to the arduino level would involve a considerable amount of infrastructure, libraries, sandbox environment, etc. It would be great to have something like that for this platform.

David

Vassius
Posts: 25
Joined: Sun Jun 03, 2012 7:56 pm

Re: Bootloader Questions

Thu Dec 06, 2012 8:15 pm

I don't get it; why do you question the usefulness of an optional piece of free software, when you apparently know the answers to your own questions and have the skills to do it better yourself?

smccain
Posts: 19
Joined: Tue Dec 04, 2012 6:51 pm

Re: Bootloader Questions

Thu Dec 06, 2012 8:37 pm

I'm not questioning anything. Please read everything I wrote. I wasn't picking on any one bootloader in particular. In fact I said "many articles have been written here." Where do you see the questioning of the implementation there?

Here I thought the AVR folks were high wound :ugeek:

Hopefully the community can get some more greybeards and less neckbeards because with "helpful" responses like that I'm sure you will liven up the day of anyone who is struggling with this stuff.

@dwelch Yes, I realize the amount of work I have in front of me. The good thing is that the arduino stuff is gcc so porting, while not trivial, won't be too much of a headache. Five years of porting mobile titles from BREW to J2ME and back should provide me with the skillset for that.

The libraries are actually the least of my worries. Getting the build environment straightened out is the first goal for me. I've already got a command line program that reads an xml file that describes projects much the same that Arduino does. I've created a small GUI front end for that and I've got the first inklings of an Arduino type environment going. I've also decided to take the winavr approach of making everything configurable. Arduino expects it's gcc to be in a certain directory (and in fact, installs it for you), but winavr doesn't care where it's at and you can point to it anywhere.

Just for clarification, I wasn't trying to denigrate any bootloaders. They are all good examples (especially dwelch's site) and they do what they need to do. I just so happen to need just a little bit more and was hoping to reach out to the community for some advise/guidance and instead I get attacked. Oh well.

Have a nice day! :mrgreen:

Vassius
Posts: 25
Joined: Sun Jun 03, 2012 7:56 pm

Re: Bootloader Questions

Thu Dec 06, 2012 9:09 pm

Well, you did write
Here is where I start to question the process
and
I fail to see the usefulness of that.
Can you see why someone might interpret that as questioning the usefulness of the existing bootloaders? ;)

dwelch67
Posts: 961
Joined: Sat May 26, 2012 5:32 pm

Re: Bootloader Questions

Fri Dec 07, 2012 1:42 am

you probably already know but the leaflabs folks have already replicated the arduino apis on an arm platform (maple) or at least that is what their literature indicates (using apis is not my thing as anyone can see from my examples).

Are you after replicating the exact arduino experience as in using the same apis, (some) applications on arduino will run as is (source code) on your platform? Or are you after the arduino like experience but new apis and such?

David

smccain
Posts: 19
Joined: Tue Dec 04, 2012 6:51 pm

Re: Bootloader Questions

Fri Dec 07, 2012 3:30 am

@Vassius Sure, you can cherry pick and come to any conclusion you want. :mrgreen: Let's suffice to say that was not my intention and I sincerely apologize if you did think that and was offended in any way. I'm not here to put down, I'm here to learn and maybe make something that will give back and help others. That's all. I think that's why we are all here :D

@dwelch67 Yes, I knew about leaflabs as well as zduino (?) the fpga core with arduino like support (I have a DE0 Nano, Papillio-One board, and a Spartan 3E breakout). If their stuff is open source I might take a look there and see what they have. I think api's have their place, especially for people new to the technology. There is a lot of things going on with microelectronics that can be intimidating for beginners and I think the Arduino api's help newcomers with that. When, and if, they decided to "dig deeper" then they can shed things like digitalWrite and digitalRead which takes hundreds of clock cycles (because it has to support so many variants) for direct PORTA and PORTB access.

The ARM itself is also way more complex than any AVR. They are real processors capable of running real software with threading, process protection, and different execution states and, as such, they are just as complex as any modern microprocessor.

I am going to abstract that with an environment that supports ease of use for beginners and flexibility for power users who want to use it as a tool to build programs for the ARM.

I will need a three main things to accomplish this goal. The first is a gcc that I can use to compile the code. Code Sourcery is being a bit protective of their open source stuff (now that they are making an IDE) so I decided to build my own toolchain.

The second thing I need is a set of api's and a standard build environment to satisfy the requirement that it's easy for users. I'll take a look at the maple stuff like I said above and if not I'll just work on porting some of the existing AVR libraries to ARM (though I recall seeing someone else claiming as such in a post somewhere - I'll have to search around).

The third thing I need is to develop some standards to describe projects to the system (which I've done - it's xml) and a core system to build those files (which I've done). I also went ahead and cross compiled make for mingw so real power users can just drop to the command line and run rt-make (the name of my make executable).

I don't want to reinvent the wheel, just use the spokes, rim, and tire that someone else built to build my own wheel.

This will be completely open source and will be hosted on github.

tufty
Posts: 1456
Joined: Sun Sep 11, 2011 2:32 pm

Re: Bootloader Questions

Fri Dec 07, 2012 5:50 am

You have a problem : existing build systems are too complex.
Solution : roll your own build system. You now have 2 problems.

Seriously, use something that exists already. The best in the long run is probably autotools.

smccain
Posts: 19
Joined: Tue Dec 04, 2012 6:51 pm

Re: Bootloader Questions

Fri Dec 07, 2012 8:26 am

tufty wrote:You have a problem : existing build systems are too complex.
Solution : roll your own build system. You now have 2 problems.

Seriously, use something that exists already. The best in the long run is probably autotools.
The build environment is only part of my solution, although I will take a look at autotools and see if can get them to work. They look like they might need cygwin though and I don't really want to have to force beginners to download a bunch of tools they probably won't use (because the tool abstracts that). The idea is to abstract the complexity of the build environment, not make a new build environment, much like Arduino abstracts away the whole process of gcc, as, ld, etc... A user just makes a program, implements a startup and loop function plus any associated functions, hits a button and she can see her program run. That's the experience I'm trying to recreate and I don't think autotools will give me that "out of the box", but it might be a tool (pardon the pun) in my efforts to achieve that.

Also, please note, this is not a problem I have. I am not trying to solve my problem, but I'm trying to lower the barrier of entry. I'm sure everyone on this board would appreciate lowering the barrier of entry and make it possible for people who may be otherwise intimidated by the process get involved with building baremetal applications for this powerful architecture.

tufty
Posts: 1456
Joined: Sun Sep 11, 2011 2:32 pm

Re: Bootloader Questions

Fri Dec 07, 2012 5:51 pm

Okay, then go for something that doesn't require a working commandline - Wrapping something around Jam would work, or you could do it using what's built into eclipse (not that I like eclipse but it certainly allows setting up a relatively moron-proof environment). You might even be able to do something using the arduino environment itself. Developing another toolset is a really bad idea, though.

smccain
Posts: 19
Joined: Tue Dec 04, 2012 6:51 pm

Re: Bootloader Questions

Fri Dec 07, 2012 7:55 pm

Well thanks for the vote of confidence. I'll continue to pursue my bad idea, thank you very much. Imagine if all the innovators in the world listen to the curmudgeon detractors? :roll:

How about a reply on my original question regarding the bootloaders? I'm not asking advise on my project (otherwise the title would have been Build System Project and not Bootloader Questions - even though this conversation has deteriorated to several of you tearing apart everything I've said - thanks for the warm welcome guys). I mentioned it so you can see where the impetus for the bootloader comes from. It's also complete (sans an installer and some integration pieces), so there is no worry about me wasting my time (which is so considerate of you :| ).

Hopefully I can make something that a begginer can use on windows. Eclipse is not a beginners tool and it requires extra steps that a beginner might get intimidated by. Reusing the Arduino environment is one approach, but I chose another.

Have a nice day. :ugeek:

tufty
Posts: 1456
Joined: Sun Sep 11, 2011 2:32 pm

Re: Bootloader Questions

Fri Dec 07, 2012 8:26 pm

It's perfectly feasible to write a bootloader that listens to rs232 and flashes a new image if it hears something. I'd probably add a hardware triggering aspect if I was doing that, pulling a pin to gnd on boot, for example. It's going to be quite a lot more work than anything that exists already, though I'd guess Dave's stuff would be a really good starting point.

My "vote of confidence",as you put it, is caused by the bitter experience of having been lost down that particular rabbit hole myself. You can, of course, listen to advice, or ignore it completely. That amount you're invoiced for it is the same. As for the "warm welcome", you've had one, and are still getting one. You might be wearing it out, though.

tritonium
Posts: 79
Joined: Tue Jan 03, 2012 7:10 pm

Re: Bootloader Questions

Fri Dec 07, 2012 11:34 pm

Hi
It sounds as if what you are proposing is what I am looking for but not knowledgeable enough to implement.
Dexos has been doing some ineresting work http://www.raspberrypi.org/phpBB3/viewt ... 72&t=16860
His idea is not unlike yours but his approach is (I think) to take an assembler (fasm) and creat macros to give "BASIC LANGUAGE" commands and I must say it looks very promissing. Of course this has the benefit of 'getting into' assembler gently.
At the moment there are only a few functions as I think he has got bogged down getting file handling (SD and FAT) and USB reliable as he is only relying on the 'system' for graphics and booting - initially anyway, and has been quiet for a while, but I expect him to pop up with an announcement any day now. (no pressure Dex!) :D
Dexes OS (DexOS) http://www.dex-os.com/DexBasic/DexBasic.htm creates a kernel.img for the SD card and once programmed is permanent until overwritten. It runs on power on.
But this creates a nightmare of SD card changing when developing code, and so using Dwelches bootloader makes development much more pleasant experience. Put it on the SD when finished.
Of course when we get file handling to SD then this will likely change.
Dave H

dwelch67
Posts: 961
Joined: Sat May 26, 2012 5:32 pm

Re: Bootloader Questions

Sat Dec 08, 2012 4:53 am

I dont see why you couldnt read/write the sd card as your non volatile storage and have the user experience be no different than a microcontroller with on chip flash (and some flavor of on chip bootloader). It is a matter of talking to the sd card and adding files and writing and reading them, non-trivial but obviously doable.

toolchain: I like to think that I demonstrate that it is possible to write somewhat portable code that doesnt care what flavor or version of gcc you are using (or llvm/clang for that matter). Also I have a set of build scripts in another one of my repos (build_gcc) for building the latest binutils, gcc, llvm and clang for various targets. I bounce between the two codesourcery flavors, plus my own all the time. I found this one recently but have not tried it:
https://launchpad.net/gcc-arm-embedded
have used emdebian as well.

Another path you might consider, which I am more and more convincing myself that is the right path, is to get the user to buy two raspberry pis. One can natively compile and with the few changes I outlined somewhere in my repo, connect to the target raspi using serial with two or three wires. This gives you among other things 1) the users are using a known linux and not a mixture of windows and linux (mostly windows). 2) A native toolchain gcc instead of arm-whatever-gcc. 3) host computer serial interface that mates up with the target raspi loader. The price of a second raspi is between on par and double what one might pay compared to the device they are going to need to pay for in order to use the serial bootloader anyway.

There is no technical reason why you cannot do what you are trying to do, it is just a matter of manpower to get it done.

The one negative thing I will say is that it is not a trivial thing to obtain the arduino experience. It is more than polish and apis and web pages, there is something emotional there that pulls homebrew and hackers into it. You can see the mbed and the launchpad stuff from ti, luminary with the stellaris before they became ti. St has good platforms, code go with them but not the same traction, rabbit semi if anyone remembers them, microchip has been trying to get win this experience longer than even the avr folks, but for whatever reason folks flock to avr freaks and then later to the arduino. (I think microchip HAD that userbase, but they migrated or a new userbase formed that surpassed the pic lovers. Microchip completely failed (and still do to this day) to understand the affordable eval platform concept). If you could duplicate that user experience and as a result user base, there would be corporations lined up around the corner to beg, borrow, or steal it.

As delivered the raspi basically gives video, audio, usb/keyboard/networking for a fraction of the price (and significantly higher horsepower, although higher actual power consumption as well) as the arduino equivalents. The raspi suffers on the gpio side, and it is hard to touch the overall atmel documentation and ability to find the documentation experience, the raspi vendor docs are almost as bad as it gets. I have said this before, I would love to see the user base take over the documentation and at least have it in one place and consistent rather than bouncing around between the vendor doc, the web based errata, and user examples that explain "what they really meant was".

If nothing else the bootloader you are proposing will be popular with the folks already able and willing to do bare metal.

David

dwelch67
Posts: 961
Joined: Sat May 26, 2012 5:32 pm

Re: Bootloader Questions

Sat Dec 08, 2012 4:58 am

To repeat my prior answer to the bootloader question. We do not have access to on chip or on board non volatile storage. The removable sd card is your nv storage, so you need to communicate to the sd card, have code that can handle the file system, and add a file or files to manage the image you want the board to run when it is not running the bootloader. It is a lot more code than you would need for a microcontroller bootloader, but at the same time, if you were to come up with such a feature, think about how many different boot images you could store on the card (too many to manage in a simple bootloader).

David

Bakul Shah
Posts: 321
Joined: Sun Sep 25, 2011 1:25 am

Re: Bootloader Questions

Sat Dec 08, 2012 9:44 pm

Vassius wrote:The bootloader obviously isn't intended for persistent programs. It's intended for use during the development phase, so that you won't have to copy your program/kernel to the SD card every time you have an update to evaluate.

If you want it persistent, just replace the bootloader with your program on the SD card.
I think what the OP wants is to save on reload time, if loading a large program. During debugging you may need to work on the same image multiple times as you are zeroing in on a problem. One idea is for the bootloader to wait for a certain amount of time after reboot, looking for a magic pattern on the uart. If it is not seen in that time, to load the program previously saved on the SD card. If the pattern is seen, to allow a new program to load in memory and save a copy on the SD card. If there is working SD card code, this doesn't seem like very hard to implement. This is something I would have used in the days when ethernet was not easily available. Now I just use uboot to tftp load a kernel from a fileserver -- this takes about 30 seconds from reboot (but you do need to setup a dhcp server, tftp server etc. which is not trivial).

benzn
Posts: 10
Joined: Mon Dec 03, 2012 12:48 am

Re: Bootloader Questions

Sat Dec 08, 2012 10:40 pm

Bakul Shah wrote:Now I just use uboot to tftp load a kernel from a fileserver -- this takes about 30 seconds from reboot (but you do need to setup a dhcp server, tftp server etc. which is not trivial).
Eeash, why does uboot take so long? Are you sending over a full linux kernel? Loading from a remote fileserver (as in on the internet somewhere)? All of my bare metal bootloaders (at least the part that changes - kernel.img) are tiny. I've been tempted to set up u-boot paired with a VM that has all the associated things set up (toolchain, tftp, etc) but not so tempted if it will take longer than the swap in/out SD card stuff

Return to “Bare metal, Assembly language”