[RANT] do we know what we are doing and why?

[hippykin]

Why do we do it?, Are we kernel developers or operating system developers? Or don't we know the answer to any of these?
I have been interested in operating system development for around three years. However progress in writing my own operating system has been slow due to exams, during the years 2007 to 2000 I did write what I consider to be a functional kernel, it can execute user programs in ring 3, can load modules, modules could very easily mount a new file system (there was an interface for this). Eventually the kernel reached a point of completion and I began writing drivers however I soon found this tedious -the fact of the matter is these days you need to implement USB and SATA - both pretty boring and complex specifications, so I began thinking that maybe I was only interested in operating system development because of the challenge, I must admit that the development of my kernel was challenging I had to program in a ways I never programmed before. When it got to the point where I had to begin developing drivers I soon found that I was not enjoying myself and that the logical thing for me to do would be to find a new hobby.
What I have realised since developing my kernel is that I can understand most of the LINUX kernel's source, so now I'm thinking of starting a new operating system project where I take a relatively recent LINUX kernel and build a new file system and GUI on top. I don't really know why i want to do this I guess it's because I have always wanted to develop a operating system however the stage that I've got to at the moment is very boring (in my opinion) so I am now going to work on a new aspect of operating system development: the user interface, I'm hoping that I will enjoy this.
What my experience so far has made me wonder is this: how many people who start operating system development actually know why they want to do it, I know that some believe they can out code Microsoft or GNU - I am not one of them, others code and OS or kernel for purely academic reasons (the L4 kernel is a prime example of this), others do it to the challenge. I am interested in your reasons for operating system or kernel development.
From looking around the project pages on this site I get the impression that not many make it to having a GUI, less make it to having a GUI with useful programs and I suspect that very few (if any) have a GUI, a network driver for a least one card along with user programs that make use of network access along with a functional USB driver (supporting at least flash memory sticks).
I do acknowledge that many developers here are only interested in text mode - however to be honest (and I mean no offence to anyone here) if I wanted an operating system with a GNU C compiler port, a NASM port and a bash port I could just use a LINUX distribution.
I wish that I had been around 18 years ago (no SATA, no USB) once one had a kernel you probably only had to write an IDE driver to support 99% of computers, probably only had to write one or two graphic card drivers etc.
All hobbyist activities evolve for example in 1976 Steve Wozniak built the Apple one personal computer - and by built I mean he created the address bus, the data bus, the memory controller, the teleprinter and the tape drive control circuitry. Nowadays there is no such thing as a computer hobby - you wouldn't dream of trying to create a motherboard for an Intel core two Duo in your shed. - the activity has moved on.
I am postulating that perhaps the same thing has happened to hobbyist operating system development i.e in 1991 Linus Torvalds was able (single handed) to write a kernel impressive enough (and with enough potential) to attract other developers to project - and we all know the rest of that story. At the moment (in 2009) no one person (or small groups of peoples) could possibly develop anything impressive enough to promote large scale interest in other developers.
If you were building a shed you would not consider ‘making’ the wooden planks yourself (by this I mean cutting down and processing trees) you would not consider building the lock or the hinges in fact you would probably buy the whole thing as a kit from someone else.
I am wondering if anyone else agrees with me that now the kernel should be considered a ‘block’ that is to say something you don't build yourself you take a existing kernel say the LINUX kernel or the BSD kernel and you alter it slightly and use it in your operating system – assuming you use the LINUX kernel your still free to write your own file system and drivers, your own kernel API, your own library and your own GUI, however you can use any LINUX kernel module that already exists meaning effectively your operating system can easily be adapted to use any hardware which LINUX supports.
I can see from other projects on this site that this philosophy already exists in the area of boot loaders most of the impressive operating systems developed by members of this site use GRUB as a boot loader (there are some exceptions) - I myself spent a month trying to write a boot loader however once I understood everything the boot loader had to do I was happy to use GRUB. - I think most of the people here see GRUB as a ‘block’ or black box.
Note I still believe that you should write a kernel if you're interested in how the low-level stuff works I just think it is ridiculous to use it underneath a serious attempt at an operating system.

To summarise:

Is the kernel aspect of hobbyist operating system development dead?
Should popular open source kernels be seen as starting points for hobbyist operating systems?
How many of you are interested in just kernel development?
Why do you do it - what is the attraction? - And yes I know this is answered in the wiki, however in my opinion it is not answer very well and I've never seen that page change significantly in three years.

I am very interested in your views - as I am trying to justify my decision to ditch two years worth of work on my kernel - thank you.

[NickJohnson]

To summarise:

Is the kernel aspect of hobbyist operating system development dead?
Should popular open source kernels be seen as starting points for hobbyist operating systems?
How many of you are interested in just kernel development?
Why do you do it - what is the attraction? - And yes I know this is answered in the wiki, however in my opinion it is not answer very well and I've never seen that page change significantly in three years.

I am very interested in your views - as I am trying to justify my decision to ditch two years worth of work on my kernel - thank you.

I don't think the kernel aspect of hobby operating system development is dead at all. On the contrary - almost all of the projects started here start as a kernel or even bootloader, then build from there.

The kernel is, although not an operating system by itself, the sort of soul of one. FreeBSD and Linux both use a lot of the same tools and libraries in their base system, but can you really consider them to be the same just because of that? The features and support the kernel provides inevitably shapes the way the rest of the system grows. Not to mention that, due to licensing and development issues, the perfect kernel for a planned system is not always available. What if you come up with a feature that requires kernel support, but you don't want to wade through the shifting sands of Linux or completely tear through a smaller kernel to get it working? Nearly all free kernels are (modular) monolithic (Linux, BSD), or extremely microkernel (l4, Mach). What if you want a specific hybrid?

At least personally, I'm not in it exclusively for kernel development, but I think it's an integral part of OS development, and one of the more fun pieces. But I'm just as happy to get the kernel working and start sculpting userspace. For me, the fun is in writing *the whole system* from start to finish. It's a hobby - why do we need a reason to do it?

Edit: Also, I'm not too worried about driver development, because I'm planning to write a wrapper that uses Linux modules as microkernel-style drivers to get obscure hardware working. That makes the whole kernel hurdle a million times smaller.

[skwee]

Well as for me, its just a hobby.

Back around 10-12years ago, I had no clue about computers. My first PC run DOS with MC and I played robocop on it (and Wolfenstein).

Back 6-7 years ago, when I first learned C, I had no clue what happens when you press "F5 - compile". For me it was magic, how my words are converted to machine language that it can understand. Today I know that the process is pretty simple.

I my self tried few topic in development, one of them that I really liked was GameDev. But I always found my self implementing the low level stuff (Before even using DirectX or OpenGL, I wrote my own 3D renderer based on some library that gave me features of 2D, to admit my 3D renderer was sucks, but how cares I wrote it ^^ ). Also in this area you are pretty limited, at some point rendering cubes will become boring and in order to render something real you either should know to use some 3D modelling tool, or have a modeler in your team.

Reading Linus's book and watching movies like "Pirates of silicon valley" makes me sorry a bit that I wasn't born 10-20years before my actual birth date. Today its not interesting, you can't really invent something new, and truly almost no one writes OSes those days (I mean a pure OS from zero), most of them use at most Linux (modifying it) or vxWorks. And IMHO it sad, cause its really interesting.

Now back to the reason why I do it. As I mentioned in the first post, just because its fun. You really understand what is going on inside the big (not so big those days) box that stand on your floor. And for now I couldn't find something that is more interesting for me than writing low level code, be it an OS, or some driver. I also hope that my hobby will help me in future to understand Linux's code easier, and to be able to leave my mark in history, however this is not what driving me to wake up in the morning/back from college and in my free time to code my OS, I don't think of money or something like this, just hobby, and the feeling you get when you done implementing X and it works as you wish!

Also you mentioned about using ready stuff:

I do acknowledge that many developers here are only interested in text mode - however to be honest (and I mean no offence to anyone here) if I wanted an operating system with a GNU C compiler port, a NASM port and a bash port I could just use a LINUX distribution.

The point is not to port as much software as possible, but its to write an OS. This is the reason I use GRUB because I want to write kernel not boot-loader. If I wanted to write a boot loader I would take Linux/Unix/*bsd and try to load it with my boot-loader, if I wanted to write drivers I again would take one of the OSes I mentioned, compile their kernel and write drivers for it, however this is not my aim.

This was my 1cent.

[Troy Martin]

It's far from dead. We do it cause a) it's fun and b) we can.

I do acknowledge that many developers here are only interested in text mode - however to be honest (and I mean no offence to anyone here) if I wanted an operating system with a GNU C compiler port, a NASM port and a bash port I could just use a LINUX distribution.

I'd say that's just a very ugly blanket statement. For example, I'm not "only interested in text mode," it's just that I feel that for the current stage of my OS a command line shell in standard 80x25 VGA text mode works best. If someone wants to go to the lengths of coding a VGA or VBE driver so I can have text mode in <long number> by <same long number * 0.75> resolution or whatever, they can go ahead and do that. If they want to build a custom GUI toolkit on top of that and make a sample graphical shell, they can do that. I don't really think it's necessary right now.

I have always wanted to develop a operating system however the stage that I've got to at the moment is very boring

I pretty much stopped reading your post thoroughly there. Operating system development is generally boring when you're writing low level code. It may be hard work, but it's really fun when you're at the stage where you can port programs and you add random crapola features into the kernel just for the hell of it.

OSDev isn't all fun and games. It's hard work, but it's worth it.

It's also fun when you make the people in your computer class go "holy crap how'd you make that!?" when you demo your kernel.

[hippykin]

I am not attempting to be mean by asking this question, it will help me clarify my thoughts (i am still trying to decide weather or not to carry on with my kernel, or to start writing higher level stuff on top of the BSD, or Linux kernel)

Assume your OS gets to this stage:

kernel complete – i.e you can run a reasonable large number of programs simultaneously, they can dynamically allocate memory access hardware and read and write files.

User interface to a point where the system is usable.

all the programs you as the developer see as critical ported or completed.

What do you do now?


Note;
i am beginning to think that i have made a mistake in assuming everyone here is interested in operating system development purely to learn, it would seem from some of the responses that some are not just interested in learning alone (but rather for the quest to write an OS from scratch) – i cant criticize that

to clarify,

i think i have learned all i can from writing a kernel, sure i have a lot more to learn about 3rd party hard where, i could learn how every different flavor of realtech's sound cards work (or at least exactly how the kernel says 'play this stream') but it doesn't interest me.

[earlz]

Edit: Also, I'm not too worried about driver development, because I'm planning to write a wrapper that uses Linux modules as microkernel-style drivers to get obscure hardware working. That makes the whole kernel hurdle a million times smaller.

Wow...that's possibly the greatest idea I've ever heard... course, I bet this is a bit harder than it sounds lol

[hrniels]

I'm also developing my own OS since one year now. Like the other said, too, I do this just because it's my hobby. It's great to really know what is going on behind the scenes. I think you learn more about many aspects in computer science than by any other project. Its nice to experiment which ideas or implementations work (well) and which don't. And its a great challenge.

But as it seems, I'm more interested in developing a userspace than many other people (if I look at other hobby-OSs). My goal is to develop a whole operating system on my own, and IMO the userspace is definitely an important part of it. But my goal is not to have an operating system that is usable productivly some time. So I do not plan to port user-apps to it. Although I don't want to exclude that I might add a POSIX-compatibility-layer, if thats possible
But the kernel is of course the most important part of it, so I spent most of the time with it. The user-apps and the (incomplete) libc exist mostly to test the features of my kernel, have a nice UI the users can play with and because I do enjoy writing them.

Should popular open source kernels be seen as starting points for hobbyist operating systems

This is of course subjective and depends on the goal you have. For me this was never an option because I want to do as much as possible on my own. I mean, this is the whole point of doing OS-dev as a hobby, isn't it? If I wouldn't want to do this work I could simply use Linux and be happy with it...
But as I said, this depends on your goal. If somebody doesn't want to build a whole OS or even kernel but just try out an idea for a specific part of it it might make more sense to use a popular open source kernel as a starting point.

[Solar]

<ThreadBranched By="Aurdal" Title="Wrapper For Other OS Drivers To Use In Hobby OS">
http://forum.osdev.org/viewtopic.php?f=15&t=20936
</ThreadBranched>

My own contribution to OS Dev is not another OS project, but the PDCLib. It's a project of limited, well-defined scope, and something that can be done in hobbyist terms.

Why do I do it?

Most of my day work is legacy code maintenance, documentation work, and lots of red tape. Being able to work with a very sharply defined specification, and to work towards my own quality standards, is like vacation for me.

In the office, where estimations are whittled down by management and necessary improvements aren't done because they are "too expensive", I am forced to work with, and ultimately release, sub-par code and documentation. Not only does it go contrary to my own work ethics, in the end the higher maintenance costs far outweight what was saved in the beginning. But that is a different cost center, so it's OK with management. That kind of thinking makes me sick, and hobby work like the PDCLib is my counterweight to it. My anchor in sanity. My rememberall of what software engineering should be about.

The fact that it can be a valuable building block for other OS projects is something that makes me proud, but not my motivation for doing it.

[Cognition]

Personally I think the primary problem overall is expecting that one person can or will do all of the work. Once you get to a certain point you should be focusing on designing a good kernel API, driver interface, and excellent documentation to go with both of them. With that people can contribute drivers and write their own software. I think a lot of people really kind of ignore or take for granted the role a developer community and user base plays in getting successful software and hardware started. To elaborate on your example, Wozniak did a lot of work himself in terms of design and the first few Apple models. Once the Apple 2 was started though a lot of the production had to be contracted out as it was just too much work for one man to do by hand. Likewise the platform didn't really take off until it had the BASIC interpreter wired in to it and third party software developers piled in. What really makes market entry difficult these days though is the expectations of consumers, Apple could sell a computer that would cost around 3500 USD today and would only run 4 applications. Obviously the average household or business would ask a lot more of a computer these days. There are however new less demanding markets especially in various portable devices or embedded type applications.

Still I think you could have pretty large potential user base by supporting common standards among computers today. Good design and implementation will always garner more attention. For all their strengths both Linux and Windows tend to suffer from a number of draw backs due to their different approaches to certain issues and the philisophical underpinnings of the groups that control their development. Human beings might have discovered fire ages ago, but people are still designing better light bulbs today. Personally I just like to tinker around with various ideas, I have some doubts that my kernel will ever be released in a usable form even at a text mode level. Not because I couldn't, but because it primarily serves as a personal playground for me and there just isn't much of an emphasis put on it.