Kernel Design - Opinions wanted

[BASICFreak]

As I stated int the title I do encourage opinions, as most of kernel design is opinion anyways...

So, unfortunately I haven't worked on code in 4-5 days due to not being able to make up my mind on where to go.
I have a working, mostly, x86 kernel currently Monolithic - and I have realized I have a "huge" decision to make, How do I want my kernel to work. Most my ideas are unorthodox and my thoughts are sometimes random - Hope you can follow this.

Using Code box to hopefully organize a little

1.)

Code: Select all

Continue with the Monolithic Kernel Design and implement all "Common" drivers (VGA, PS/2, FDD, HDD, etc...) and allow drivers for other devices or replacements be installed via System Int to make the the PIC to the code needed for IRQx

Added drives will still be in Ring3 except for the IR which will be running in Ring0 when called from IDT allowing use of privileged commands and full HW control to an extent

Yes this can cause a security hole, which can be slightly patched by signing

2.)

Code: Select all

Remove all drivers from my Kernel and only have the bare core for the CPU to run and make a pure MicroKernel.

This would involve creating a stage 3 loader to open a file (text or fs like binary) listing all drivers and relative IRQ# and Load into memory, all while keeping track of it on a table in memory - possibly a temporary IDT that the kernel can just move and map. OR writing in pure ASM and expanding stage 2 (or better yet finding a way to link 16 and 32 bit together with LD)

And as before can still add drivers via System Int

The only downfall I see with a MicroKernel is the kernel loader must be a kernel itself handling IRQs and ISRs and FDD (& HDD) drivers due to limitation in real mode on memory.

---------------

And part two MultiTasking Resource Management and INTs

I have no task or thread management, yet, I have TSS setup just enough to get to Ring 3 and I currently have my kernel mapped with Ring 3 access - will be changing this soon.

I by no means want HW Task Switching due to wanting this to run on a larger range of systems, so I will be implementing this in software linked to the PIC (currently set at ~100Hz)

This is mostly going to be about Resource I/Os.

I believe I understand how linux is handling all inputs and outputs, but please correct me if I'm wrong. Linux handles most if not all devices attached to the system as a file available for read and/or write (such as /dev/tty0 for the first terminal and /dev/sda for the first HDD) and the kernel handles whether the task has permission to access the file or not (also R/W).

I do see why it was done this way and how it simplifies the API, but I like being weird so here are a couple of my ideas

1)

Code: Select all

Have a shared buffer in memory for each device driver installed with input or outputs and make the API handle everything

The location of buffer can be got from a Sys Int

2)

Code: Select all

Having a shared buffer for device drivers and local buffer for keyboard and video and have the kernel decide where the K/B data goes and which video buffer to use

3)

Code: Select all

Have a System Int for each installed driver and require a system call for all inputs outputs etc.

4)

Code: Select all

My favorite one, Create a Driver struct in the kernel and allow all inputs and outputs to be mapped into said table and kernel controls which task is linked or can use the device.

This is similar to Linux in a way, but it would be separate from the VFS and would have a few more options (at least in my own opinion)

Yes it was a long post... but I cannot do any work with this rolling through my head constantly. And this is not even half of it but I will not bring it up at-least yet.

This was real brief and not very detailed all I'm interested in is the opinions of my fellow OSDEVers to help me make a decision so I can get back to work on code

--

If your opinion does not match anything above I would still love to hear it, but PLEASE keep this topic Civil - don't get mad if someone else's opinion differs from your own or standard practices.


- Brian

[bwat]

So, unfortunately I haven't worked on code in 4-5 days due to not being able to make up my mind on where to go.
I have a working, mostly, x86 kernel currently Monolithic - and I have realized I have a "huge" decision to make, How do I want my kernel to work. Most my ideas are unorthodox and my thoughts are sometimes random - Hope you can follow this.

If you had a specification, then you wouldn't have this problem. However, as a hobby developer you may just be on a random journey seeing where this project takes you - opening yourself up to serendipity - and in this case I would go with what you think would be most enjoyable. I've done this myself in the past but I'm not sure I'll do it again. I specify, in one form or another, all my hobby projects now and I've come to believe that there's truth behind the saying that form is liberating.

[Brendan]

Hi,

So, unfortunately I haven't worked on code in 4-5 days due to not being able to make up my mind on where to go.
I have a working, mostly, x86 kernel currently Monolithic - and I have realized I have a "huge" decision to make, How do I want my kernel to work. Most my ideas are unorthodox and my thoughts are sometimes random - Hope you can follow this.

You might want to decide (and/or tell us) what the main goals for your OS are. For example, if you really want to focus on things like security, fault tolerance, minimising downtime and/or (soft or hard) real-time then a micro-kernel may be more suitable; but if you want to focus on max. performance or minimising development time then monolithic may be more suitable.

The only downfall I see with a MicroKernel is the kernel loader must be a kernel itself handling IRQs and ISRs and FDD (& HDD) drivers due to limitation in real mode on memory.

No. Typically your boot code would load some sort of "boot image" (a file containing many other files - e.g. kernel, VFS, file systems, device manager, drivers, etc) into memory using the firmware. Once that's done you'd setup/start the kernel (from RAM), setup/start the VFS (from RAM), setup/start the device manager (from RAM), etc. Eventually everything you need to access normal file systems (on disk) have been started, and you can switch to loading files from a file system.

Of course for both monolithic (where kernel can grow to 5 MiB or more) and micro-kernel (where "boot image" can grow to 5 MiB or more), you will need to be able to load more than 640 KiB of stuff from disk. This is normally done by repeatedly switching between real mode (to call BIOS) and protected mode (to copy data loaded by BIOS to a higher address) until everything you need is loaded wherever you want it.

For the remainder of your post, I see 2 issues. First, I get the impression that you don't quite understand the role IPC plays in operating systems. For example, with any type of IPC, the kernel's IRQ handler can use it to send "something" (e.g. maybe a signal) to a CPL=3 driver without causing a security hole. For a micro-kernel IPC has an very strong influence over the way software communicates with device drivers.

Secondly; a major part of a modern OS is some sort of "device manager". This is code that detects which devices are present (e.g. using things like scanning PCI bus) and starts the corresponding device drivers (and manages resources, etc). Don't forget that this device detection doesn't just happen during boot - a lot of hardware is "hot-plug" now. For example, every time you insert or remove a USB device something is going to have to determine what that USB device is and find/load the appropriate device driver.


Cheers,

Brendan

[BASICFreak]

If you had a specification, then you wouldn't have this problem. However, as a hobby developer you may just be on a random journey seeing where this project takes you

You pretty much hit the nail on the head... I came to OSDEV from PHP and JAVA applets just to learn a couple of languages and I haven't ever been able to learn the easy way, but the main reason I'm here is to say I have done it and to say I have an OS with 90+% my code (still some segments I have to rewrite from back in the tutorial following days)

You might want to decide (and/or tell us) what the main goals for your OS are. For example, if you really want to focus on things like security, fault tolerance, minimising downtime and/or (soft or hard) real-time then a micro-kernel may be more suitable; but if you want to focus on max. performance or minimising development time then monolithic may be more suitable.

My current idea is mostly based on boot speed, performance, and stability (of witch I have 1.5/3 as I'm not fully stable and performance wise I could use a boost but there is no such thing as over optimizing...)

I'm looking to have as little processing overhead as possible on kernel side so other tasks can have the most CPU time.

First, I get the impression that you don't quite understand the role IPC plays in operating systems.

And that impression would be correct - I have a very brief understanding of it, as with most things OS related, I'm learning as I go and this is the next avenue. Anyways thanks for the link, looks like it will be very useful.

Secondly; a major part of a modern OS is some sort of "device manager". This is code that detects which devices are present (e.g. using things like scanning PCI bus) and starts the corresponding device drivers (and manages resources, etc).

I realize this and by no means was I tiring to say that I wouldn't have this.


--- Slightly Off-Topic ---
In a multi-tasking environment where all processes are linked lets say at 0x200 (I know) and I have lets say 5 processes I would have to update the Virtual Memory on every task switch in order for any pointers to work, correct?

I ask because I'm going to get these threads up tonight hopefully, and get my kernel back to kernel only memory - and I do not want to assume things too often.

[Brendan]

Hi,

You might want to decide (and/or tell us) what the main goals for your OS are. For example, if you really want to focus on things like security, fault tolerance, minimising downtime and/or (soft or hard) real-time then a micro-kernel may be more suitable; but if you want to focus on max. performance or minimising development time then monolithic may be more suitable.

My current idea is mostly based on boot speed, performance, and stability (of witch I have 1.5/3 as I'm not fully stable and performance wise I could use a boost but there is no such thing as over optimizing...)

Performance points towards monolithic, while stability points towards micro-kernel.

For boot speed it doesn't matter so much. The important things here are:

• do whatever is most important before doing things that are less important (e.g. if the most important thing is to get the disk device driver running, then you don't want that to take longer because the sound card driver is using all CPU time doing some sort of lengthy self test
• do as much as possible in parallel. Ironically, a lot of well known OSs don't do this - they'll initialise the disk controllers (and let CPUs be idle while waiting for hardware delays), then once that's done they'll initialise the network card (and let CPUs by idle while waiting for hardware delays), and so on.

--- Slightly Off-Topic ---
In a multi-tasking environment where all processes are linked lets say at 0x200 (I know) and I have lets say 5 processes I would have to update the Virtual Memory on every task switch in order for any pointers to work, correct?

Yes. Typically this means changing the value in one register (CR3) whenever a CPU switches from a thread in one process to a thread in a different process.


Cheers,

Brendan

[BASICFreak]

Thanks for all the input.

So now my idea is a monolithic (hybrid maybe) that will have default drivers for all IRQ devices built in; and PCI device drives loaded from disk in the background (no PnP for now, maybe soon).

And I am still going to be debating API for Ring3, but for now I can setup system Ints for basic IOs and add on to it in the near future when I'm truly ready for it (when I understand it better need to read up on it).

- Brian

[AndrewAPrice]

The only downfall I see with a MicroKernel is the kernel loader must be a kernel itself handling IRQs and ISRs and FDD (& HDD) drivers due to limitation in real mode on memory.

No. Typically your boot code would load some sort of "boot image" (a file containing many other files - e.g. kernel, VFS, file systems, device manager, drivers, etc) into memory using the firmware. Once that's done you'd setup/start the kernel (from RAM), setup/start the VFS (from RAM), setup/start the device manager (from RAM), etc. Eventually everything you need to access normal file systems (on disk) have been started, and you can switch to loading files from a file system.

GRUB supports loading modules for multiboot kernels. That's actually how I use to test user-space loading without having file system drivers.

[BASICFreak]

The only downfall I see with a MicroKernel is the kernel loader must be a kernel itself handling IRQs and ISRs and FDD (& HDD) drivers due to limitation in real mode on memory.

No. Typically your boot code would load some sort of "boot image" (a file containing many other files - e.g. kernel, VFS, file systems, device manager, drivers, etc) into memory using the firmware. Once that's done you'd setup/start the kernel (from RAM), setup/start the VFS (from RAM), setup/start the device manager (from RAM), etc. Eventually everything you need to access normal file systems (on disk) have been started, and you can switch to loading files from a file system.

Switching to PMode then back to RMode over and over seems very costly, it seems simpler to just rewrite a mini driver.

GRUB supports loading modules for multiboot kernels. That's actually how I use to test user-space loading without having file system drivers.

Yes but GRUB unfortunately breaks my FDC driver, or should I say my FDC driver is so unstable GRUB makes it not work.




P.S. Just finished my first thread/task manager currently just working with 2 threads Idle and Kernel so the processor has a load around 50% of the time and I can switch between rings.

All my code tested and ran on real h/w better than bochs


-Side Thought-
Still need to fix my i8042 init and FDC driver so they work "correctly"

[Brendan]

Hi,

The only downfall I see with a MicroKernel is the kernel loader must be a kernel itself handling IRQs and ISRs and FDD (& HDD) drivers due to limitation in real mode on memory.

No. Typically your boot code would load some sort of "boot image" (a file containing many other files - e.g. kernel, VFS, file systems, device manager, drivers, etc) into memory using the firmware. Once that's done you'd setup/start the kernel (from RAM), setup/start the VFS (from RAM), setup/start the device manager (from RAM), etc. Eventually everything you need to access normal file systems (on disk) have been started, and you can switch to loading files from a file system.

Switching to PMode then back to RMode over and over seems very costly, it seems simpler to just rewrite a mini driver.

The cost of switching CPU modes is nothing compared to the cost of the disk IO itself. For example, for 5 MiB of data you could load 512 KiB at a time and switch to protected mode and back 10 times. If each CPU mode switch costs 100 cycles, then that adds up to 2*10*100 = 2000 cycles, or about 2 us on an old/slow 1 GHz CPU.

Now...

"A" mini driver is nowhere near adequate. You'd need one for ATA disk, one for AHCI/SATA, about 100 for different SCSI controllers, 4 for various USB controllers, etc. However; that's only the controllers and not the devices themselves. In all of those case the devices have variations (e.g. "ATA disk" alone has several PIO modes plus several DMA modes, a collection of different features that may or may not be supported, etc) and you're going to have to deal with all of those variations (for all types of devices attached to all types of controllers). Then you're going to need device detection code (just to figure out which driver to use and which resources the device has), and you're also going to need basic stuff like memory management, timers, IRQ handling, etc. so that you don't need to build everything into every driver.

Mostly, it's not "a mini driver" at all, but it actually becomes "a mini OS". Of course then you need to figure out how you're going to load the mini OS into memory (so it can load the real OS) and you're right back where you started (unless you want "mini mini drivers" that load the "mini OS" that loads the OS!). Essentially, it's a massive amount of pointless work that does nothing more than create problems.


Cheers,

Brendan

[Rusky]

If you're willing to drop BIOS support, depending on what direction you wan to go, you can boot from UEFI and let the firmware handle it without mode switching, or you can have coreboot load a kernel+drivers as its payload.

[AndrewAPrice]

GRUB supports loading modules for multiboot kernels. That's actually how I use to test user-space loading without having file system drivers.

Yes but GRUB unfortunately breaks my FDC driver, or should I say my FDC driver is so unstable GRUB makes it not work.

What was the problem? As far as I recall, GRUB just copies the file from disk, dumps it into memory, and gives you a pointer to it. If you're using an executable format, you still need to parse it to extract an executable image.

If you're interested in keeping the file system or disk drivers outside of the kernel, this is a great way to go because users can modify GRUB at boot time and load the modules they need if you want to run your OS on some exotic setup.

[BASICFreak]

GRUB supports loading modules for multiboot kernels. That's actually how I use to test user-space loading without having file system drivers.

Yes but GRUB unfortunately breaks my FDC driver, or should I say my FDC driver is so unstable GRUB makes it not work.

What was the problem? As far as I recall, GRUB just copies the file from disk, dumps it into memory, and gives you a pointer to it. If you're using an executable format, you still need to parse it to extract an executable image.

Only thing I can guess is stability, my FDC will not read a single sector (everything returns 0's) if it is not booted with Broken Thorn's Stage 2 (Tutorial 14)... And my stage 2 is not fully operational yet...

If you're willing to drop BIOS support, depending on what direction you wan to go, you can boot from UEFI and let the firmware handle it without mode switching, or you can have coreboot load a kernel+drivers as its payload.

Unfortunately all my test systems are '96-99 which if I did my research correctly would not have a UEFI - first introduced in 2007?.

Coreboot looks interesting, but I still rather the PC load up on my own code.

[Rusky]

Coreboot is an open source firmware. You would end up running less third-party code with it than without it- you flash it + a payload into the boot ROM and that gets run straight from there. You could use the same driver code in your kernel and the boot process, with no mini-driver in the firmware or your bootloader- either by running the kernel from ROM (makes upgrades harder) or by making the bootloader be the ROM kernel's init process.