My idea for the loading from disk of a FS driver

[earlz]

Ok, whenever I made my OS, I had a consistency problem. I wanted for all of the drivers to be outside of the actual kernel, making a microkernel/exokernel...

Well, I ran into the problem of loading the FS/Disk driver... Afterall, how do you load the driver that loads? Well, I came up with what I think is a pretty smart idea, and can make it so your OS can boot from most any storage device and filesystem. (without recompiling or patching)

my idea is quite simple.
the bootsector loads 3 files, rather than 1. First it loads the disk driver to a constant address, then it loads the filesystem driver to a [different] constant address, and finally, it loads the kernel.
The kernel now has the filesystem and disk driver in memory, already loaded, so it can now use them and load other drivers and such...

what do you think of this idea?

[Telgin]

Actually, I think that's fairly close to what most OSes do these days. I think that's how Windows does it at any rate. NTLDR (I think, could be some other part of the OS) loads what are known as boot drivers before loading the kernel to its specified address. This is usually just disk controller / filesystem drivers, but could be more I suppose. That way, whenever the kernel is loaded, it can immediately use the drivers to load other parts of the OS.

I think once you get to the point in your OS that you are actually using drivers in addition to a simple single kernel file, this is going to be the simplest and most practical way to do things.

[deadmutex]

Well I don't see any other real way of doing it... The whole point of the bootloader is to initialize the system so that it has a suitable environment for the kernel and drivers to run. Since a bootloader isn't a kernel, you could have any driver built into it, really. Another option is to use the initial bootloader to load a larger bootloader that can load arbitrary drivers and pass real-mode boot information to user servers.

[bewing]

Well, it kinda depends on exactly what you mean by a "file". When you make a device bootable, it needs to have a kernel placed on it, and whatever drivers you need for the device. The process of making a device "bootable" can cram everything you need for that specific device into one "file", without having to rebuild the kernel or anything. Your kernel just has to be able to find the drivers -- wherever the bootloader loads them. So it is possible to do it all in one file if you want -- even if you are staying away from a slightly-monolithic design of just having a singletasking driver built into the kernel that only runs for just a little while and then is discarded.

[Brendan]

Hi,

The problem is installing the OS, rather than booting an installed OS.

Typically you'd want a generic boot CD that can be used to boot any computer (and then install the OS or use it as a "rescue disk"). In this case you don't know which device driver/s the kernel will need to access the CD-ROM - it could be ATA/ATAPI, one of many different SCSI drivers, or a USB controller driver and USB storage device driver.

For similar reasons you might not know which kernel you'll be using. For e.g. the previous version of my OS had 6 different micro-kernels (where the boot code decided which one to use), and the new version has many "kernel modules" (several linear memory manager modules, several scheduler modules, etc, where the boot code decides which modules to use).

Once you start looking at 30 or more seperate files, making the boot loader loader each seperate file gets messy. It's better for the boot loader to load a single "boot image" that contains many files.

Of course once you're using a boot image, it makes sense to put anything you might want (rather than anything you might need) into the boot image to speed up boot times (e.g. so things like the GUI could be started without waiting for disk drivers and file systems to become ready).


Cheers,

Brendan

[quok]

What I do is pretty similar to what linux 2.6 does. I have an initrd which I have grub load as a module. That initrd is a simple CPIO archive. I have kernel code to read a cpio archive and use it as a filesystem. In fact, that's the only filesystem code I have right now, but the plan is to use that CPIO archive the same way linux uses it -- to store all the boot drivers needed to get the kernel up and running.

That way, an installation cdrom would just use a different CPIO archive which contains all the "popular" drivers.

[Ready4Dis]

Wow, i typed a lot, and it got lost... SOB. Well, not typing it all again, but what you say is similar to what I do:

http://www.osdev.org/phpBB2/viewtopic.p ... ght=#84937
Loads kernel + up to 3 drivers, 2 of them being file system and disk i/o. I split mine into 2 sectors, first sector is file system and media type specific, second sector uses first sectors read_block function to read the rest of the info, but is completely media/FS independant. It is responsible for setting up a20, pmode, virtual memory, etc. Once the kernel loads, this memory is considered free, so there is no code laying around uselessly . Between the 2 of them, I use 1024 bytes, but really not even that much. My application assumes the kernel + 3 drivers are linear files and is given a start location (upon format of the media). I currently have support for fat12, fat16, fat32, and a custom file system, as well as floppy and IDE support (and drivers in my kernel). Once the kernel loads, it can do what it needs.

[cyr1x]

Another way would be to load the Filesystem first which then loads the kernel and jumps to it.

[Ready4Dis]

Another way would be to load the Filesystem first which then loads the kernel and jumps to it.

Yes, that is also a way, but I like loading my kernel + memory manager, FS driver and i/o driver all at once, i initialize my memory manager first, then my i/o and fs drivers before continuing on. This means my FS driver can allocate memory, etc, no special case situations it needs to be programmed for. Now if I loaded a FS & disk driver first, they would have to have special sections of code just to handle booting up, which is no longer used once loaded. Almost 100% of my unused code is discarded when my bootloader jumps to my kernel, and no special code needs to be done in any driver besides the memory manager, which needs to assume some memory is already allocated to store the kernel + drivers, but that is a simple parameter that stores the size of each driver, and is part of the memory manager's initialization, so no special cases if the file sizes of the other drivers change or anything like that.

[Ready4Dis]

Hi,

The problem is installing the OS, rather than booting an installed OS.

Typically you'd want a generic boot CD that can be used to boot any computer (and then install the OS or use it as a "rescue disk"). In this case you don't know which device driver/s the kernel will need to access the CD-ROM - it could be ATA/ATAPI, one of many different SCSI drivers, or a USB controller driver and USB storage device driver.

For similar reasons you might not know which kernel you'll be using. For e.g. the previous version of my OS had 6 different micro-kernels (where the boot code decided which one to use), and the new version has many "kernel modules" (several linear memory manager modules, several scheduler modules, etc, where the boot code decides which modules to use).

Once you start looking at 30 or more seperate files, making the boot loader loader each seperate file gets messy. It's better for the boot loader to load a single "boot image" that contains many files.

Of course once you're using a boot image, it makes sense to put anything you might want (rather than anything you might need) into the boot image to speed up boot times (e.g. so things like the GUI could be started without waiting for disk drivers and file systems to become ready).


Cheers,

Brendan

Installing an OS is no problem at all, even from a CD. Since the bootable image is treated as a floppy disk (well, similar), you have access to any files on that drive. From there you simply load whatever other drivers you want, i don't see how that is any different than how loading it another way would go. Generic booting is just as easy this way, so I don't see the argument there.

Yes, a generic boot CD, so, your first major drivers will be.... a FS for the boot image, and i/o driver for the boot image. Then from here, load all the other drivers you require.

I see no point in having more than 1 kernel, my kernel doesn't do much really, so design differences are key here I think. It wouldn't take much at all for my boot loader to be able to load more than 1 kernel, just store a list of kernel locations, and load whichever is deemed 'best'. Pretty basic. However, as said, my kernel is pretty minimal, my user apps and drivers do most of the work.

I also disagree with using a boot image to cover 30 different files. Why in the world would my boot loader load 30 different files? Once it loads the FS and i/o driver, the OS takes control and the boot loader is no longer required. The system can then search and load any drivers it deems necessary. What if I don't want to run the GUI, but text only, why am I loading a vesa driver from a boot image? I like my boot loader to load the bare minimum (which for my OS is fs, i/o and memory manager, because i wanted to be able to make changes to my memory manager without changing the kernel out, although this will probably end up as part of the kernel once i'm happy with it, at which time I will use my spare loaded driver into a process handler for my multi-tasking, and replace my crappy one in my kernel until I am happy with that, and move it back into the kernel).

I don't even see the point of putting the GUI as part of your image, what if I have a specific driver for my video card, you're telling me it's better to add that to the image and load that before my fs/io drivers? That just doesn't make much sense, read it off the disk, so when I want to do updates, it's just a matter of replacing a few files rather than re-packing a kernel image. Lets be honest, the speed difference between you loading a generic vesa driver from your image, then the fs and i/o drivers, then the specific video card drivers is going to slow down your boot process rather than speed it up as claimed. What good is a GUI if you haven't loaded anything from disk yet to display, unless you're loading a humungoid OS image from the disk which has all the images, icons, text, backgrounds, etc built in.

[Brendan]

Hi,

Installing an OS is no problem at all, even from a CD. Since the bootable image is treated as a floppy disk (well, similar), you have access to any files on that drive. From there you simply load whatever other drivers you want, i don't see how that is any different than how loading it another way would go. Generic booting is just as easy this way, so I don't see the argument there.

You're right - I forgot this would only be used while you're still in real mode (and you'd have real mode file system code during boot and seperate protected mode file system code used after boot).

I see no point in having more than 1 kernel, my kernel doesn't do much really, so design differences are key here I think. It wouldn't take much at all for my boot loader to be able to load more than 1 kernel, just store a list of kernel locations, and load whichever is deemed 'best'. Pretty basic. However, as said, my kernel is pretty minimal, my user apps and drivers do most of the work.

For me, the boot code detects some things (CPU features, physical memory, NUMA domains, etc) and then decides what is needed. For example, it can check if all CPUs support long mode and then use the 64-bit linear memory manager. If long mode isn't supported it can then check if 36-bit addressing is needed (and if PAE is supported) and then use the "36-bit" linear memory manager. Otherwise it'd use the 32-bit "plain paging" linear memory manager.

My previous OS had 6 kernels (single-CPU, SMP and NUMA; with and without PAE), but that was when the OS was "32-bit only". I've split the kernel into modules now (several linear memory managers, several schedulers, etc).

The idea is to automatically select the best kernel (or the best kernel modules) without the user doing anything. For a generic OS installation CD all kernels (or kernel modules) would be present for the boot code to choose from, but when the OS is installed you'd only install the kernel (or kernel modules) that are needed for the specific computer.

I don't even see the point of putting the GUI as part of your image, what if I have a specific driver for my video card, you're telling me it's better to add that to the image and load that before my fs/io drivers? That just doesn't make much sense, read it off the disk, so when I want to do updates, it's just a matter of replacing a few files rather than re-packing a kernel image. Lets be honest, the speed difference between you loading a generic vesa driver from your image, then the fs and i/o drivers, then the specific video card drivers is going to slow down your boot process rather than speed it up as claimed. What good is a GUI if you haven't loaded anything from disk yet to display, unless you're loading a humungoid OS image from the disk which has all the images, icons, text, backgrounds, etc built in.

The GUI was only an example - I was thinking more of getting the GUI's log-in prompt up, so the user can start typing in their username and password without waiting for disk drives to become ready (which takes Linux more than 10 seconds on the computer I'm using).

In a more general sense, the idea is to do things during boot in parallel as much as possible, instead of doing nothing while waiting for the disk drivers, then doing nothing while waiting for file system checks, then doing nothing while waiting for the keyboard driver, then doing nothing while waiting for the network cards, then doing nothing while waiting for DHCP, etc. Part of this is not needing to wait for disk drives and file systems to start before anything else can start. However, even for "wait then wait then wait" style serial boot it's probably not a bad idea to load everything you need from contiguous sectors to minimize the number of disk seeks (including reading directory information from disk).

I should also point out that there's differences between different OSs. For some OSs (during boot) there's a large number of small delays and a small number of large delays. For other OS's the hardware initialization is "half-assed" and a lot of these delays are avoided.

For a simple example, some OSs will detect if a PS/2 controller is present, then test the PS/2 controller, then configure the PS/2 controller, then detect which PS/2 devices are present, then test the PS/2 devices, then configure the PS/2 devices, then start using the PS/2 devices. Other OSs will assume a PS/2 controller is present, then assume the PS/2 controller is working correctly, then assume the PS/2 controller is configured, then assume the first PS/2 device is a keyboard, then assume the keyboard works correctly, then assume the keyboard is configured, then start using the keyboard. When something like the keyboard's BAT (Basic Assurance Test) is meant to take between 500 ms and 750 ms it's fairly obvious that the "half-assed" code will be much much faster (no hardware delays at all), but it's also very fragile and inflexible.

Let's assume the OS's code isn't "half-assed", and it takes 10 seconds to initialise the SCSI controller and drives, 4 seconds to setup networking (ethernet device driver, DHCP, etc), 3 seconds for the video driver to check for faulty display memory, and 1 second to initialize the keyboard. Would you wait 10 seconds for the drives, then 4 seconds for networking, then 3 seconds testing for faulty display memory, then 1 second waiting for PS/2 devices; so that it takes 18 seconds before the user can do anything? Alternatively, would you start the SCSI, network, keyboard and video drivers at the same time so that the user can start logging in after 3 seconds (instead of 18 seconds) and so that everything is ready after 10 seconds (instead of 18 seconds)? What if the OS has a distributed file system (and redundancy) and can access files on a remote computer and doesn't need to wait for the disk drives, so that everything is ready after 4 seconds (instead of 18 seconds)?


Cheers,

Brendan

[Andrew275]

I use GRUB's builtin module support to load my filesystem and disk drivers. It works really well, it's just a matter of going through a very simple structure that GRUB passes to your kernel that tells you where each driver is located in memory. From there, it of course depends on how your driver support works.

[Ready4Dis]

Hi,

Installing an OS is no problem at all, even from a CD. Since the bootable image is treated as a floppy disk (well, similar), you have access to any files on that drive. From there you simply load whatever other drivers you want, i don't see how that is any different than how loading it another way would go. Generic booting is just as easy this way, so I don't see the argument there.

You're right - I forgot this would only be used while you're still in real mode (and you'd have real mode file system code during boot and seperate protected mode file system code used after boot).

I see no point in having more than 1 kernel, my kernel doesn't do much really, so design differences are key here I think. It wouldn't take much at all for my boot loader to be able to load more than 1 kernel, just store a list of kernel locations, and load whichever is deemed 'best'. Pretty basic. However, as said, my kernel is pretty minimal, my user apps and drivers do most of the work.

For me, the boot code detects some things (CPU features, physical memory, NUMA domains, etc) and then decides what is needed. For example, it can check if all CPUs support long mode and then use the 64-bit linear memory manager. If long mode isn't supported it can then check if 36-bit addressing is needed (and if PAE is supported) and then use the "36-bit" linear memory manager. Otherwise it'd use the 32-bit "plain paging" linear memory manager.

My previous OS had 6 kernels (single-CPU, SMP and NUMA; with and without PAE), but that was when the OS was "32-bit only". I've split the kernel into modules now (several linear memory managers, several schedulers, etc).

The idea is to automatically select the best kernel (or the best kernel modules) without the user doing anything. For a generic OS installation CD all kernels (or kernel modules) would be present for the boot code to choose from, but when the OS is installed you'd only install the kernel (or kernel modules) that are needed for the specific computer.

I don't even see the point of putting the GUI as part of your image, what if I have a specific driver for my video card, you're telling me it's better to add that to the image and load that before my fs/io drivers? That just doesn't make much sense, read it off the disk, so when I want to do updates, it's just a matter of replacing a few files rather than re-packing a kernel image. Lets be honest, the speed difference between you loading a generic vesa driver from your image, then the fs and i/o drivers, then the specific video card drivers is going to slow down your boot process rather than speed it up as claimed. What good is a GUI if you haven't loaded anything from disk yet to display, unless you're loading a humungoid OS image from the disk which has all the images, icons, text, backgrounds, etc built in.

The GUI was only an example - I was thinking more of getting the GUI's log-in prompt up, so the user can start typing in their username and password without waiting for disk drives to become ready (which takes Linux more than 10 seconds on the computer I'm using).

In a more general sense, the idea is to do things during boot in parallel as much as possible, instead of doing nothing while waiting for the disk drivers, then doing nothing while waiting for file system checks, then doing nothing while waiting for the keyboard driver, then doing nothing while waiting for the network cards, then doing nothing while waiting for DHCP, etc. Part of this is not needing to wait for disk drives and file systems to start before anything else can start. However, even for "wait then wait then wait" style serial boot it's probably not a bad idea to load everything you need from contiguous sectors to minimize the number of disk seeks (including reading directory information from disk).

I should also point out that there's differences between different OSs. For some OSs (during boot) there's a large number of small delays and a small number of large delays. For other OS's the hardware initialization is "half-assed" and a lot of these delays are avoided.

For a simple example, some OSs will detect if a PS/2 controller is present, then test the PS/2 controller, then configure the PS/2 controller, then detect which PS/2 devices are present, then test the PS/2 devices, then configure the PS/2 devices, then start using the PS/2 devices. Other OSs will assume a PS/2 controller is present, then assume the PS/2 controller is working correctly, then assume the PS/2 controller is configured, then assume the first PS/2 device is a keyboard, then assume the keyboard works correctly, then assume the keyboard is configured, then start using the keyboard. When something like the keyboard's BAT (Basic Assurance Test) is meant to take between 500 ms and 750 ms it's fairly obvious that the "half-assed" code will be much much faster (no hardware delays at all), but it's also very fragile and inflexible.

Let's assume the OS's code isn't "half-assed", and it takes 10 seconds to initialise the SCSI controller and drives, 4 seconds to setup networking (ethernet device driver, DHCP, etc), 3 seconds for the video driver to check for faulty display memory, and 1 second to initialize the keyboard. Would you wait 10 seconds for the drives, then 4 seconds for networking, then 3 seconds testing for faulty display memory, then 1 second waiting for PS/2 devices; so that it takes 18 seconds before the user can do anything? Alternatively, would you start the SCSI, network, keyboard and video drivers at the same time so that the user can start logging in after 3 seconds (instead of 18 seconds) and so that everything is ready after 10 seconds (instead of 18 seconds)? What if the OS has a distributed file system (and redundancy) and can access files on a remote computer and doesn't need to wait for the disk drives, so that everything is ready after 4 seconds (instead of 18 seconds)?


Cheers,

Brendan

I know what you mean, but I don't really have much of a problem the way I do it, it loads VERY fast, and multi-tasking starts as soon as the kernl is loaded, so i can start loading everything in parallel after the kernel is loaded, and file system and disk i/o are initialized. I see what you're saying about multiple kernels, but right now I load my memory manager as one of my primary modules, so if I wanted to detect or support long mode, it would be there rather than in the boot loader. I just don't like the idea of my boot loader doing so much work, i am currently at 1024 bytes, just because i have it abstrated for 2 sectors to abstract the file systemdevice dependant section in the first sector. My kernel is VERY basic, so adding a 64-bit memory manager is loading another module, so to me it's not an issue, like I said kernel design has a lot to do with what sections perform what function. My OS with GUI doesn't take long at all to load, as the file system driver is very simplistic FAT implementation, multi-tasking is initialized right when the drivers are registered with the driver handler, so from that point on I can parallel load whatever. I don't have any time issues with loading my OS right now, if I run into them I might think about changing, but even on older hardware my OS takes less than 5 secods to load into the GUI, which inclues ide, mouse, keyboard, vesa, virtual memory, multitasking. It doesn't do a whole lot, and I am re-writing the majority of it, so we'll see how it ends up when i'm done. So where you have one huge image that gets loaded at bootup with 6 kernels included, i load a single one, and only load modules as needed. I beleive they will both run pretty close to the same speed since i'm doing less disk access but more processing. Also, for me to update a driver is very simple, just update the pointer to the file in the boot sector to the new driver. No updating/patching a kernel image, so trying new things is very easy, as well as reverting back . I do very little detecting in the boot loader, just memory detection, enagle a20, initialize paging, memory map lower 1mb, map kernel + drivers to it's location and the go 32 and jump to kernel (passing it a few things like memory, boot disk/partition, etc). I am rei-writing a bunch of things, and the way i handle drivers is one of them, I am going to be switching to static memory addresses for drivers, same as my user apps, so I don't have to relocate/patch them, and they can assume whatever they need. Down side is more context switches, so i'm still not settled yet, but my driver model allows for either way without much issue.

Yes, initializing SCSI may take a little while, as does detecting ATAPI (about 2.5 seconds on average from my experience), but once that's done, my multi-tasking kicks in, and I can load the rest of the drivers/system in parallel. The cool part of this, is I can put as much as I want into any of the drivers, or use the spare section for my GUI once done, so it starts initializing the GUI while loading the HD/FS driver, or replace my kernel file with an image or similar, and load everything at once, no changes to my bootloader are required for this, just update the file pointers, say maybe a stub that reads the modules instead of a kernel (gets called from boot loader), stub reads image, seperates/uncompresses files, and jumps to the kernel and gives it a list of drivers/locations that are there, barely any changes to my kernel, and no changes to the boot loader, very updateable and can easily modify for future changes.

[egos]

Hi, I'm using exactly this idea for a long time. But all my 1st stage boot loaders load only two files (small kernel and boot device/file system driver for PM). Floppy version is free for using and changing. It might be as well for you to change word "june" in the file names, which the boot loader loads. You can use this boot loader for loading the 2nd stage boot loader and its configuration file as well.

The size of june.os is a non-zero multiple of 1 Kb. The base address is 0x8000. This file ends with the signature 0xAA55.
The size of june.fs is a non-zero. ax:cx will contain this value, when the boot loader passes control to the kernel.

[bewing]

For a simple example, some OSs will detect if a PS/2 controller is present, then test the PS/2 controller, then configure the PS/2 controller, then detect which PS/2 devices are present, then test the PS/2 devices, then configure the PS/2 devices, then start using the PS/2 devices.

However, it is also important to note a few other things -- that Brendan pointed out on other threads. We can expect an OS to run for weeks, months, or perhaps years between reboots. Hardware/memory/etc. can fail while the machine is running. Yes, it is important to detect the hardware, so that your drivers work properly. But there is no good reason to test hardware during boot (beyond just testing that it exists/responds). Testing, as everyone agrees, takes lots of time; it needs to be done on a schedule in the background while the machine is up; it should not be done during a time when the machine is committed to a high priority task like booting.