A good plan?

[jammmie999]

So what do you think of my development timeline/plan? Is it feasible/practical or too ambitious, have I missed anything or made any critical mistakes? How long do you think this will take to implement?

Milestone 1: Basic Services
Kernel can provide:
• Memory Management Services - both physical and virtual
• Virtual File System - With FAT32 support, Serial ATA (SATA), and possibly SD card mount/unmount/read/write/cache capabilities - definitely ability to read/write FAT32 file
• Device Management - PS/2 Keyboard, Mouse - no USB support at the moment though, very complicated (so I am told)
Will provide a basic text mode VGA output only, this will only be temporary though

Milestone 2: Userspace
Kernel is able to switch to user space, load a file via its VFS and execute as a process. This may require reading/processing PE/ELF headers?
Processes are loaded into isolated memory space.
A basic scheduler is implemented - possibly only voluntary yield, or interrupt driven.
User space processes are able to make system calls, to print terminal text for example.
Processes may be able to communicate with each other via IPC
We could now modify our device management/detection to load a device driver in kernel/user space to deal with hardware, 'core' drivers remain compiled into kernel for performance (e.g. PS/2 Keyboard, SATA etc.) with compilation support to add/remove them as appropriate
Can be port the GCC compiler and some other basic tools.

Milestone 3: Connected
Ethernet driver interface, Ethernet drivers.
Basic TCP/IP stack - can I 'steal' this from Linux without violating GPL, or do I have to write my own
IPv4/v6 stack - again can I 'steal' this?
Application Layer Support - HTTP
Possibly basic cryptography services implemented in kernel - MD5, SSL, AES via instruction set?
Ensure we are able to establish ourselves on the network, grab IP Address from DHCP, broadcast our NETBIOS name, ping other devices, etc.
Download a file over HTTP, save to disk using our VFS.
Possibly basic 802.11 support for wireless ?

Milestone 4: Getting Graphical
Basic VBE video driver that I can use to fall back to should no other driver be available.
Define graphics card interface, can I mimic Linux and adapt there large driver base?
Write graphics card driver.
Allow userspace applications to define there graphical space, handle and pass events (such as mouse clicks, keyboard) to userspace processes.

Maybe even basic sound/audio support?

Milestone 5: Bulking out
Device detection - add support for USB
VFS - add support for USB Hard drives, additional file systems NTFS, EXT. The less common file systems shouldn't be loaded by default to avoid bloating the kernel rather implemented as kernel extension drivers.
Create a userspace process for exploring hard drive (such as explorer.exe, finder etc.)
Add some unit tests.
Add support for some more 'obscure' hardware, (e.g. Barcode scanner, RFID Reader, Card Scanner, Webcam etc.) even if this is just a defined interface with no real implementation, virtualise some of these things for testing.

Thanks

[NickJohnson]

A few notes:

You're going to need at least interrupt handling before writing drivers (and you'll probably want some form of multiprocessing/threading.)

You need to port (or write) some sort of C standard library in order to do userspace stuff. Also, you really want preemptive (not voluntary) multitasking in userspace.

You can't really 'steal' any components of the TCP/IP stack from Linux, unless you want to license your own project under the GPL as well. Of course, why miss out on the fun of writing a TCP/IP stack?

From what I've heard, accelerated graphics drivers are hard. Just getting an SVGA framebuffer isn't too bad, but you will need to implement VM86 for that. If you just want "graphics", you won't have a problem, but writing a graphics architecture with good performance could turn into an endless project.

Beyond that, it seems like a reasonable enough plan. How long it will take to implement depends on your experience with the tools and hardware, and quantity of free time. If I had no other side projects going, I'd probably be able to do all that in 18-24 months, but I have a few years of experience doing this stuff.

[bewing]

It looks like it's enough to get you started, and get you enthusiastic. Since those are the two main points anyway, it's good enough. Within 6 months, your plan will look totally different than this, and that's fine. The OS needs to know a tremendous amount of information about the system and all its components. Locating, parsing and storing all that info in memory in a sensible way will turn out to be a much bigger effort than you realize -- I call it "initializing the kernel".

Most of your points here involve accessing hardware. This all comes under the heading of "drivers". The design of your driver interface is going to be one of your biggest nightmares/tasks. There are many possibilities, and hundreds of thousands of lines of code to write to even begin to tackle that entire concept. Once that is done, all of these hardware issues you have listed will just be details.

To "steal" driver stuff from Linux, you would need to adopt a tremendous amount of the design of the Linux kernel. You would be locked into a monolithic kernel for starters. Linux code is also extremely painful to read. You would probably need to grab enough code that you would indeed fall into the GPL license. -- But you may well decide that the Linix driver interface sux, and then you won't be able to use a single line of code.

In general, I've always found that it improves my design the most when I try to push on into a stage of OS coding that I'm not ready for, yet. Such as trying to get a "Hello World" usermode app running while I'm in the earliest stages of OS design.

Your milestones will turn out not to be linear. You will certainly find that you need to do all of your milestones all at the same time. It is all interdependent. You can't design your driver interface without knowing what your TCP/IP and USB stacks will look like, and you can't design your TCP/IP and USB driver stacks without knowing how your driver interface works. That kind of circularity is what makes this whole thing such a misery.

I hope you understand that you've got at least a couple of years of full-time work ahead of you.

[Antti]

Don't take my word too seriously because I have implemented not more than the half of the things you listed. I haven't touched the network at all yet.

Why I am posting anything at all? Well, I want to say that your plan is suggesting that you are very keen to organize things. Even if this specific plan turns out to be overkill, I have a feeling that you go far with you OS developing. No joking, I seriously mean this.

However, I have different approach to make plans for these kind of projects: unless you are a very experienced OS developer, I think that it is not sensible to make too extensive plans at first. I think that these projects should have one milestone at first. The usermode app printing "Hello World" would be a perfect goal. When that is reached, you know more and can set another milestones. I think that almost everyone agrees with me.

Linux code is also extremely painful to read.

I am really relieved that you said this. Is it the general opinion that the linux code is a reference of the best practises?

[Combuster]

Linux code is also extremely painful to read.

I am really relieved that you said this. Is it the general opinion that the linux code is a reference of the best practises?

The only good practice found in linux code is that it's open source. My general sentiments about the linux code is that most of it is designed after being coded, and comments being for wussies.

[bewing]

I am really relieved that you said this. Is it the general opinion that the linux code is a reference of the best practises?

Heh. Far from it. Linux was written by amateur volunteers with one or two years of programming experience. It is full of bugs, factual errors, mistaken assumptions, bad algorithms, and just plain fugly code. Why do you think they have to post new versions of everything every 6 hours?

The only reason it hangs together at all is that they had Unix for a template. The only reason it competes with "professional" operating systems is that 40-year-old Unix was way ahead of its time.

[Owen]

Linux seems to work on the principle of infinite monkies:

1. Implement feature
2. Improve feature
3. Feature isn't fast enough? GOTO 1 & reimplement feature from scratch

Seriously. Nothing else can explain why, by now, they're on their third USB stack, second device file system, second implementation of an initrd...

Thats not to say that everything Linux is flawed. Eventually, they seem to get things right; and, certainly, they do occasionally come up with brilliant ideas:

• Futexes are a brilliant idea. I have a blog post on them.
• The design of their tmpfs/initramfs system is ingenious
• I'm sure many other things that I've forgotten.

My general view, however, is that Linux is great for ideas, less great for implementation. Some aspects of it are artifacts of history which, really, aren't good design; they're just how things always have been.

[NickJohnson]

Of course, looking at it another way, the existence of Linux is proof that if you have enough manpower, you can beat out well-designed systems with comparatively little planning. Just because it is less efficient doesn't mean it's less effective. The problem, of course, is that none of us can harness Linux-scale hordes, so looking to Linux for design ideas should always be done cautiously. Then again, how many other fully-fledged, documented, high-performance open source *nix systems are there to look to?

[Antti]

This Linux viewpoint sounds a very interesting topic. I'll start to search what is already discussed in this forum. I don't want to turn this offtopic. However, when I think this more in-depth, this is not actually very much offtopic. Well, I already started this post so here it goes:

Some people easily forbid criticism for someone how is not capable to do the thing being criticized better than the original author.

- "How are you to say that the Thing(tm) is poorly designed and implemented?"
- "Well, according to my studies and experiences it might be that way and..."
- "What? Look at how succesful it is? Isn't that telling you something? Are you saying that you can do it better?"
- "No, what I meant was that..."
- "You have no competence to say anything."
- "But..."

Fortunately, this forum does not seem to have this attitude.

When it comes to Linux, I surely can't consider myself being a kernel hacker. Of course, I have read some books of its structure and also all the interesting early public debates of it (for example, Tanenbaum-Torvalds debate). I basically want to do my own thing, being it worse or not. Sometimes I just happen to glance the Linux source code just to compare some specific implementation with my own. I stricly avoid copying anything (a legal or moral aspect) because I haven't yet decided whether to go to GPL or not.

I have had a belief that I am probably too unintelligent to understand "the creation of masters". It must be good because it is the best. Even if I found it being a mess, I would be an idiot how does not understand the system as a whole. That's how I used to think before. I somehow feel happy when you criticize the Linux system because it makes me feel that I am not the only one thinking so. In this forum I probably don't have to emphasize that we are only talking about the kernel itself. Not the userspace things but that would also be a very interesting topic.

I must admit that maybe I'll end up admiring the Linux code after I have done my own trial-error phase.

[sandras]

Personally, when I first started using Linux and started learning to use the command line, I admired the system - pipes and "everything is a file" work great together, though really, not everything is a file. And yes, I also like tmpfs. One thing that I did not like, was that even a minimal configuration, the kernel is huge. I had a minimally configured kernel, which pretty much could only give me a command line interface, and I think it was some 750KB. The userland was a statically linked busybox inside a tmpfs. That's all it was. Another thing I didn't like was udev. It seems silly, because udev creates files in /dev directory according to waht it finds in /sys. To me that's just wrong - why not just implement devfs in the first place. I heard arguments like that's so they could have permissions and names of device files according to a configuration file, but I believe that's possible with a synthetic devfs too, and as far as I know, that's what they are finally doing. Then there was the thing, that when I found Plan 9, I realized, that much more things could be represented as files. On the other hand, even in Plan 9 not everything is a file.

[Jezze]

What is the timeline? You didn't mention that.

I don't know how long you expect it to take but as a comparison it took me about 3 years to go from a newbie to finish step 1 and 2 and barely started on 3. Reason it took so long is because you keep going back and redesigning stuff you are unsatisfied with. This is of course unless you already know exactly how you want everything to work which is unlikely.

[rdos]

I'm at the last stage (5), but it took me 24 years to get there. OTOH, I've not worked full-time on the project, and during some periods I've not done anything.

Just like Jezze writes, you are likely to want to redesign things in multiple steps as you go. This might both be because you come up with better designs, and because at the early stages you don't want to put excessive complexity to things when you try to make the basics work. For instance, you might not need a VFS, a device-driver model, or "handles" to begin with. Even at higher stages, you might need to change things at the first stages. For instance, I've redesigned the multitasking module several times. I actually think it is important to redesign code / interfaces that are suboptimal, otherwise you will end up with really cludgy code.