User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Linux is huge, even when small.

Fri Mar 17, 2017 6:24 am

As has already been known, I am attempting to get a decent amount of documentation built up in preperation for going off line. I have discovered that enough documentation to be useful for the basics of Linux is a giant undertaking.

As a point of comparison, to the other documentation I am getting as part of this effort:
  • All Parallax Propeller Documentation and source code (including the design of the Propeller chip) is under 600MB.
  • All documentation and source for the Raspberry Pi firmware is just under 400MB.
  • All VideoCore IV documentation I can find comes to just at 48MB.
  • Every bit of bare metal programming documentation I can find for the Raspberry Pi takes up only 200MB.
  • Very complete documentation of the Game Boy & Game Boy Color comes to 30MB.
  • Complete documentation of DR-GEM comes to 16MB.
  • Complete documentation and source of RISC OS all RISC OS modules I know of, and all RISC OS applications I can find that are open source and ARMv8 compliant comes to just under 1.5GB.
  • Complete documentation for the Atari ST/TT/Falcon and other Atari/EmuTOS based computers, including all software and source needed comes to about 500MB.
  • AND: Just barely enough documentation to be useful for Linux and Raspbian comes to at least 26GB (still adding up). THAT IS ONLY THE DOCUMENTATION. And that is barely scratching the surface of the basics.
So it seems that Linux is huge, just in documenting enough of the system to be useful for the most basic of programs, at least to anyone that writes software.

It would take many lifetimes just to understand the minimal basics of this system, it is a wonder there are not a ton of bugs and security holes, as huge as the API is there is no way for a person to debug the entire thing, and it would take hundreds of people all debugging the entire thing to even begin to do anything with.

It would take roughly 600 years to learn enough of it to call oneself a beginner in Linux, the way Linux is today, not to mention how things will expand in the next 600 years.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
Gavinmc42
Posts: 4231
Joined: Wed Aug 28, 2013 3:31 am

Re: Linux is huge, even when small.

Fri Mar 17, 2017 7:15 am

How long are you going offline for?

Over the last 6 years years I have collected Pi/Linux and software related ebooks and html web tutorials.
Just checked the server, the Bookshelf folder is 16.8GB, contains 47,899 files, 9,320 folders.
It's not current, still some new local ebooks that need moving across.

There are no data sheets, sources files or Linux/Pi OS's etc in that folder, they are in other folders.

What is really scary is I have read a lot of them and skimmed the rest.
You might just get a glimpse of why I am a big fan of Ultibo now :lol:

Lucky for us Linux is developed/debugged by thousand of people.
The kernel itself is not too bad, 4.9 is 91MB zipped :lol:

Want to learn Linux? Start with Bash, Sed, Awk, Grep.
Shell scripting on a 1GHz Pi is quickish and you can do a lot with the above.
Small distributions like piCore, which is kernel plus busybox are usable without bloat.
Plus if you start with just this minimal OS you have to learn Linux to do anything and no need to rely on other peoples libraries and bloatware :o
Just remember that Linux treats everything as a file even GPIO.
But if you get stuck for speed piCore has Micropython as part of the standard install.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
PeterO
Posts: 5357
Joined: Sun Jul 22, 2012 4:14 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 7:42 am

DavidS wrote:It would take roughly 600 years to learn enough of it to call oneself a beginner in Linux, the way Linux is today, not to mention how things will expand in the next 600 years.
Rubbish ! There are many who have progress beyond your definition of beginner in considerably less than 600 years, some are even "youngsters" !

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

Heater
Posts: 14273
Joined: Tue Jul 17, 2012 3:02 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 8:05 am

Oh good. There is lot's of documentation for the Linux kernel, the operating systems and thousands of programs built on top of it.
Memory in C++ is a leaky abstraction .

User avatar
Cancelor
Posts: 759
Joined: Wed Aug 28, 2013 4:09 pm
Location: UK

Re: Linux is huge, even when small.

Fri Mar 17, 2017 8:07 am

Just print a copy of the internet ;-)
Can't find the thread you want? Try googling : YourSearchHere site:raspberrypi.org

User avatar
Gavinmc42
Posts: 4231
Joined: Wed Aug 28, 2013 3:31 am

Re: Linux is huge, even when small.

Fri Mar 17, 2017 9:03 am

Just print a copy of the internet
That's good, should be in the snarky post. :D

Don't forget the old archived net stuff, Linux has not changed that much in 30yrs.
Well the kernel, which is now mostly drivers being added, the rest is just application stuff and X win bloat.

It all depends on the beginner.
Any expert want to own up to knowing it all?
I gave up on "knowing it" and now just use it ;)
But you have a better chance of understanding the open source basics than trying to see inside MS stuff.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24660
Joined: Sat Jul 30, 2011 7:41 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 9:42 am

There is little chance of anyone being an expert or even knowing enough, to be able to comment on every part of the linux kernel. There is simply too much of it. You would need to be an expert on ethernet, USB, displays, OS internals, memory management, x86, ARM - everything that the kernel covers. Not possible.

Which is why there are domain experts - who know a lot about a subset of the whole.

So find docs on this bits you need to know about, and use those.

Any modern OS is too complicated for one person to totally comprehend, and OS are complicated out of necessity - they have to do a LOT.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

gkreidl
Posts: 6173
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Linux is huge, even when small.

Fri Mar 17, 2017 9:45 am

The world has changed, David.
Modern operating systems are created by a kind of swarm intelligence - tens of thousands of people are working on it. And nobody knows everything about it (that's also true for MS and Apple) - it's simply impossible.
Something like Atari TOS was really simple compared to a modern OS. It took just 3 people to rewrite it completely (Kromke and the Behne brothers, and they did not have the original source code). And TOS was really buggy. Kromke found hundreds of bugs and fixed them. He offered the fixes to Atari but Jack Tramiel didn't want to spend a dime for debugging.
The source of a modern web browser is much more complex than the whole Atari OS and it depends on a lot of libraries which are not a part of the browser itself.

Look at it this way to see the advantage: If you want to write a program nowadays, 99% has already been done and is provided by the OS and libraries.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

jahboater
Posts: 4930
Joined: Wed Feb 04, 2015 6:38 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 10:16 am

gkreidl wrote:The world has changed, David.
Modern operating systems are created by a kind of swarm intelligence - tens of thousands of people are working on it. And nobody knows everything about it (that's also true for MS and Apple) - it's simply impossible.
The old mainframe operating systems were also large and incomprehensible.

I think Linux is rather special in that it fits on really tiny devices all the way through to super computers. It is not huge for what it does. How can you compare it to RISC-OS that only runs on early ARM processors?

Looking at the Pi Zero on the desk in front of me - its running a highly capable multi-user multi-tasking operating system with a full development environment and much more. Does seem miraculous for someone who grew up with mainframes and "mini" computers.

I count 326 Linux API's by the way (what DavidS would call SWI's).

User avatar
DougieLawson
Posts: 36870
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Linux is huge, even when small.

Fri Mar 17, 2017 11:21 am

jahboater wrote:
gkreidl wrote:The world has changed, David.
Modern operating systems are created by a kind of swarm intelligence - tens of thousands of people are working on it. And nobody knows everything about it (that's also true for MS and Apple) - it's simply impossible.
The old mainframe operating systems were also large and incomprehensible.
The old mainframes are still there and they're still running the world (at least I hope they are because working with mainframe databases and transaction managers is my day job). The old mainframes have also sprouted lots of Linux things which are running the new world (and acting as glueware to the old stuff).

That 50 year old S/360 OS/VS2 dinosaur ain't dead yet, it's changed a bit with 31-bit addressing (MVS/XA) and now 64-bit addressing (z/OS). The mainframe can still easily outstrip any Intel X86_64 based processing because of the highly parallel I/O subsystems. They don't even get warm while running a 100,000 transactions per second or 50 million SQL updates per day.

I've been working with IMS, that's IBM's large database & transaction manager system for 35 years and there's still bits of it that I don't know in detail (and I am a well known subject matter expert). I've been working with Linux since 1995 and it gives me something new every day.

The skill folks need is being able to analyse what they're seeing and come up with coherent solutions to their immediate problems, that's more productive than trying to learn the whole thing. When that doesn't solve your probem being able to give a good statement of your error, and your thinking behind it so that another expert can help is most important. Being willing to run the experiments and testing to get as much in the way of problem diagnostics is the next most important thing.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

jahboater
Posts: 4930
Joined: Wed Feb 04, 2015 6:38 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 11:37 am

DougieLawson wrote:The mainframe can still easily outstrip any Intel X86_64 based processing because of the highly parallel I/O subsystems. They don't even get warm while running a 100,000 transactions per second or 50 million SQL updates per day.
Yes indeed. From distant memory, the old and weedy Amdahl mainframe I used had things like multi-pathing which was a separate channel for each arm/head within a disk and I think you could send a whole screen full of text and formatting with a single instruction, the channel processors did all the work. The 5 mips (yes!) machine easily supported over 100 VM's (VM/SP) with users doing serious work.

Heater
Posts: 14273
Joined: Tue Jul 17, 2012 3:02 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 11:42 am

Given that a modern mainframe CPU looks like this:

Image

I would say it has moved on a bit since 50 years ago! Count the pins.

And they generally have a bunch of them I would hope they can turn in some good performance.

Edit: Not so modern. That image is from 19985.
Memory in C++ is a leaky abstraction .

blaablaaguy
Posts: 623
Joined: Sun Sep 27, 2015 3:26 pm

Re: Linux is huge, even when small.

Fri Mar 17, 2017 12:15 pm

Heater wrote: Edit: Not so modern. That image is from 19985.
19985? I would have thought people would at least have computers embedded in their brain by then :)
This signature intentionally left blank.

User avatar
DougieLawson
Posts: 36870
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Linux is huge, even when small.

Fri Mar 17, 2017 1:17 pm

Heater wrote:Image

I would say it has moved on a bit since 50 years ago! Count the pins.

And they generally have a bunch of them I would hope they can turn in some good performance.
Things, have moved on a bit. We've got lots of tightly coupled processor cores (individualy they're not running very fast but they're very good at running parallel workloads), some dedicated to special purposes (like running Linux and Java). No pictures have emerged from Poughkeepsie of the new power7 processors (which is what the microcoded zSeries processor is based on).

The most fun I've seen with a mainframe was the 18 year old guy who bought one (I've met him). He dismantled it and rebuilt it in his basement. That lad will go a long way as a hardware specialist. https://www.youtube.com/watch?v=45X4VP8CGtk
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 2:27 pm

Gavinmc42 wrote:How long are you going offline for?
A few months if all goes to plan. Though must also be prepared in case there is no internet access for anyone by the time I return.
Over the last 6 years years I have collected Pi/Linux and software related ebooks and html web tutorials.
Just checked the server, the Bookshelf folder is 16.8GB, contains 47,899 files, 9,320 folders.
It's not current, still some new local ebooks that need moving across.

There are no data sheets, sources files or Linux/Pi OS's etc in that folder, they are in other folders.

What is really scary is I have read a lot of them and skimmed the rest.
You might just get a glimpse of why I am a big fan of Ultibo now :lol:

Lucky for us Linux is developed/debugged by thousand of people.
The kernel itself is not too bad, 4.9 is 91MB zipped :lol:
Yes the Linux kernel is not to bad. And yes the whole system ontop of it is to much. No we are not lucky of the way it is debugged, we would be lucky if it was such that it could be understood by single people, as debugging could be more complete then.

And you may see why I am such a fan of RISC OS. Ultibo is a much more complicated OS than RISC OS.
Want to learn Linux? Start with Bash, Sed, Awk, Grep.
Shell scripting on a 1GHz Pi is quickish and you can do a lot with the above.
Small distributions like piCore, which is kernel plus busybox are usable without bloat.
Plus if you start with just this minimal OS you have to learn Linux to do anything and no need to rely on other peoples libraries and bloatware :o
Just remember that Linux treats everything as a file even GPIO.
But if you get stuck for speed piCore has Micropython as part of the standard install.
I know that much of Linux already. I prefer not to use extra libs at all.

I feel that for Linux one should know xlib, xcb, Linux (the kernel), sh, twm/vtwm, xdm, OpenGL, SDL, sockets, clib/libc, all HW related kernel modules, virtualization kernel modules, and how all of this interacts with each other and itselfs. And I do NOT mean simply knowing how make any function call and all the flags (and in the case of the UI stuff [sh, twm, etc] NOT just how to script/use), I mean knowing each component very deeply, to the point of having enough knowledge to re-implement it if you really had to.

If you could get that far you would be far enough to say that you have began to start to scratch the surface of what there is to know about the very basics of Linux.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 2:33 pm

PeterO wrote:
DavidS wrote:It would take roughly 600 years to learn enough of it to call oneself a beginner in Linux, the way Linux is today, not to mention how things will expand in the next 600 years.
Rubbish ! There are many who have progress beyond your definition of beginner in considerably less than 600 years, some are even "youngsters" !

PeterO
On that we will have to agree to dissagree.

My definition of getting far enough to be just starting, and not even a beginner with Linux would be to understand 100% completely everything about the Linux kernel, init, all kernel modules, pam, sh, clib/libc, gcc, xlib, xcb, xdm, twm/vtwm, OpenGL, sockets, and everything directly releated to these as well as enough understanding of the operation to be able to reimplement the majority of all of it (excepting the HW interface stuff). You get that far in understanding of Linux and you would be able to say you are about 1% of the way to being a beginner, as much more as there is to Linux.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 2:36 pm

gkreidl wrote:The world has changed, David.
Modern operating systems are created by a kind of swarm intelligence - tens of thousands of people are working on it. And nobody knows everything about it (that's also true for MS and Apple) - it's simply impossible.
Something like Atari TOS was really simple compared to a modern OS. It took just 3 people to rewrite it completely (Kromke and the Behne brothers, and they did not have the original source code). And TOS was really buggy. Kromke found hundreds of bugs and fixed them. He offered the fixes to Atari but Jack Tramiel didn't want to spend a dime for debugging.
The source of a modern web browser is much more complex than the whole Atari OS and it depends on a lot of libraries which are not a part of the browser itself.

Look at it this way to see the advantage: If you want to write a program nowadays, 99% has already been done and is provided by the OS and libraries.
Do you forget that Linux is supposed to be a Unixish system?

Unix was created as a system that is simple enough to understand, as the previous mainframe/mini-computer OSes were getting way out of hand, and incomprehencible do to being a swarm development style. Unix was supposed to be the embodyment of KISS.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 2:40 pm

blaablaaguy wrote:
Heater wrote: Edit: Not so modern. That image is from 19985.
19985? I would have thought people would at least have computers embedded in their brain by then :)
Nah, by then we will have succeeded at destroying this planet and be somewhere in interstellar space looking for a new home world, a few thousand years into our colony space flights (though only a few weeks time experienced by the crews do to temporal dilation of moving at speeds approaching 0.8C). :) :lol:
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

hippy
Posts: 6532
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Linux is huge, even when small.

Fri Mar 17, 2017 3:59 pm

DavidS wrote:It would take many lifetimes just to understand the minimal basics of this system
Yet people can use Linux and write programs which run under Linux with very little effort with no need to have more than a superficial understanding of Linux.

I am not sure what Linux knowledge you think you need but I suspect you are looking for far more than a regular programmer or user of Linux actually needs.

Edit: Just saw this ...
DavidS wrote:My definition of getting far enough to be just starting, and not even a beginner with Linux would be to understand 100% completely everything about the Linux kernel, init, all kernel modules, pam, sh, clib/libc, gcc, xlib, xcb, xdm, twm/vtwm, OpenGL, sockets, and everything directly releated to these as well as enough understanding of the operation to be able to reimplement the majority of all of it (excepting the HW interface stuff). You get that far in understanding of Linux and you would be able to say you are about 1% of the way to being a beginner, as much more as there is to Linux.
Well good luck with that.

To me it seem overly ambitious and rather unnecessary.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 4:07 pm

hippy wrote:
DavidS wrote:It would take many lifetimes just to understand the minimal basics of this system
Yet people can use Linux and write programs which run under Linux with very little effort with no need to have more than a superficial understanding of Linux.

I am not sure what Linux knowledge you think you need but I suspect you are looking for far more than a regular programmer or user of Linux actually needs.
Yes it is easy to learn to write X programs in Linux, it shares enough with other Unix.

Though to learn enough to claim any understanding is a different story. And as with any system the better you understand it the better you can code for it and the better results you can produce in better ways. Thus the minimum understanding level I posted twice in responses.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

hippy
Posts: 6532
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Linux is huge, even when small.

Fri Mar 17, 2017 4:10 pm

DavidS wrote:Thus the minimum understanding level I posted twice in responses.
Sorry I missed that earlier, updated my post previous as you were posting yours. As I added there it seems to me to be overly ambitious and rather unnecessary.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 4:19 pm

hippy wrote:
DavidS wrote:Thus the minimum understanding level I posted twice in responses.
Sorry I missed that earlier, updated my post previous as you were posting yours. As I added there it seems to me to be overly ambitious and rather unnecessary.
:lol:

Well it is only getting the same level of understanding of Linux that you would get with most other Operating Systems within the first week of ever using the others.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 6:47 pm

Now running on my fresh install of Raspbian, checking to make sure I have everything.

Next up will be RISC OS.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
kusti8
Posts: 3439
Joined: Sat Dec 21, 2013 5:29 pm
Location: USA

Re: Linux is huge, even when small.

Fri Mar 17, 2017 6:59 pm

I think the main issue is the definition of a beginner. You define beginner as someone who develops for and with all the resources Linux has to offer. I and most people view a beginner as someone who moderately understands the command line and some of the commonly used commands.

Think of it like university. Linux is the entire school that offers many different courses. It can do anything with that collective brainpower. Yet no one can begin to get a diploma in all the majors the university has to offer because it's impossible. So everyone studies one or a few things they are particularly interested in and do that. As a collective, all of those students in the university contribute to Linux. But they have a basic understanding of the other topics, they just don't know the specifics.

And BTW, big documentation is a good thing, it just means that everything is thoroughly documented. In a world like this where Linux must serve many different needs, the documentation grows as more features and changes are added. The appeal to me is that the kernel still stays pretty small.
There are 10 types of people: those who understand binary and those who don't.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Linux is huge, even when small.

Fri Mar 17, 2017 7:14 pm

kusti8 wrote:I think the main issue is the definition of a beginner. You define beginner as someone who develops for and with all the resources Linux has to offer. I and most people view a beginner as someone who moderately understands the command line and some of the commonly used commands.

Think of it like university. Linux is the entire school that offers many different courses. It can do anything with that collective brainpower. Yet no one can begin to get a diploma in all the majors the university has to offer because it's impossible. So everyone studies one or a few things they are particularly interested in and do that. As a collective, all of those students in the university contribute to Linux. But they have a basic understanding of the other topics, they just don't know the specifics.

And BTW, big documentation is a good thing, it just means that everything is thoroughly documented. In a world like this where Linux must serve many different needs, the documentation grows as more features and changes are added. The appeal to me is that the kernel still stays pretty small.
I see your point, though in the OS world it is a bad point.

On the documentation, yes I am glad it is all available, though the huge amount of documentation I have covers just barely enough to scratch the surface, and is concise.

On other systems every small detail of every aspect of the system is covered in a lot less, though is just as if not more complete.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

Return to “Off topic discussion”