What should I do with my OS?

[bwat]

It saddens me to see people jump on SpyderTL's invitation to original thought. I would have expected the hobbyist developer community to be more supportive. As hobbyists we aren't constrained by the user's inability to accept new ideas, or by a project's meagre budget, in fact, we're only really limited by our hardware and our imaginations. I write this as someone with 15 years of professional OS development experience.

Nobody is forcing us to choose another person's design created to meet a set of specifications that have nothing to do with our reality. You can reimplement Unix if you want, but, is that mimicry creation? Wouldn't you rather be creative?

Here's Alan Kay on epistemological stance and computer system design (it's worth it for what you can do in 10,000 lines of code):
http://www.youtube.com/watch?v=FvmTSpJU-Xc

[h0bby1]

conceptually, i would seperate the lowest level part that is mostly the kernel and drivers and base hardware ressource mannagement, and the part that is more of system libraries, and general software organisation, and higher level functionality, but they can both be told to be part of the OS to a degree, like having some linear algebra math using fast vector math like for 3D rendering or audio analysis to do all kind of thing as base system library, who can potentially also take advantage of the hardware if present and give an api or interface to make higher level use of the hardware, it's not all that clear where the OS part is supposed to end and where application level start

[rdos]

conceptually, i would seperate the lowest level part that is mostly the kernel and drivers and base hardware ressource mannagement, and the part that is more of system libraries, and general software organisation, and higher level functionality, but they can both be told to be part of the OS to a degree, like having some linear algebra math using fast vector math like for 3D rendering or audio analysis to do all kind of thing as base system library, who can potentially also take advantage of the hardware if present and give an api or interface to make higher level use of the hardware, it's not all that clear where the OS part is supposed to end and where application level start

Where the OS ends and where the application begins is only a problem in old-fashioned OS-designs where the kernel has no access to the user-mode API. At least in my model, the kernel can use any user-level API (except for synchronization where it needs to use the kernel variant), and thus a piece of code can usually be placed in both kernel and usermode at the choice of the developer. Of course, a piece of code that needs to use the kernel API must be in kernel, but for other code the developer can chose.

[rdos]

The general order of implementation is something like Memory Management -> Threading -> Scheduling -> IRQ Management -> CPU Management, then probably a unified VFS. From there you'd build the device tree using your VFS primitives and move into loading drivers, before getting a userspace running.

No, the order is Setting up debugging with exception handlers -> Memory Management -> IRQ Managament -> SMP-related debugging -> SMP support -> Task-related debugging -> Scheduling

[h0bby1]

conceptually, i would seperate the lowest level part that is mostly the kernel and drivers and base hardware ressource mannagement, and the part that is more of system libraries, and general software organisation, and higher level functionality, but they can both be told to be part of the OS to a degree, like having some linear algebra math using fast vector math like for 3D rendering or audio analysis to do all kind of thing as base system library, who can potentially also take advantage of the hardware if present and give an api or interface to make higher level use of the hardware, it's not all that clear where the OS part is supposed to end and where application level start

Where the OS ends and where the application begins is only a problem in old-fashioned OS-designs where the kernel has no access to the user-mode API. At least in my model, the kernel can use any user-level API (except for synchronization where it needs to use the kernel variant), and thus a piece of code can usually be placed in both kernel and usermode at the choice of the developer. Of course, a piece of code that needs to use the kernel API must be in kernel, but for other code the developer can chose.

for me any library or code portion that is supposed to be used by several application can fit conceptually into the os as system libraries, even if an os can also include third party libraries to provide some framework to application, it's still supposed to be part of the os bundle, if many applications need to include the same library of piece of code in order to run, it could mean the os is missing something, anything that can ease up application developpement, or allow to add easily some functionality into application can fit into the system, even if it's not directly related to kernel space

[rdos]

for me any library or code portion that is supposed to be used by several application can fit conceptually into the os as system libraries, even if an os can also include third party libraries to provide some framework to application, it's still supposed to be part of the os bundle, if many applications need to include the same library of piece of code in order to run, it could mean the os is missing something, anything that can ease up application developpement, or allow to add easily some functionality into application can fit into the system, even if it's not directly related to kernel space

By placing user-level interface code in system libraries (like the system DLLs in Windows), you effectively make sure that kernel code (like device-drivers) cannot use this interface. That's a bad design.

[h0bby1]

for me any library or code portion that is supposed to be used by several application can fit conceptually into the os as system libraries, even if an os can also include third party libraries to provide some framework to application, it's still supposed to be part of the os bundle, if many applications need to include the same library of piece of code in order to run, it could mean the os is missing something, anything that can ease up application developpement, or allow to add easily some functionality into application can fit into the system, even if it's not directly related to kernel space

By placing user-level interface code in system libraries (like the system DLLs in Windows), you effectively make sure that kernel code (like device-drivers) cannot use this interface. That's a bad design.

kernel code shouldn't have the need to use system library, system library i think like font mannagement, audio/video/image coding/filtering, some file format manngement like compression or other like xml or what's not, but if the code has to be used by kernel device drivers yes it shouldn't be placed in user space, but anyway it's mostly likely those system library would depend on the kernel directly or indirectly so having the kernel code depending on them would make a cyclic dependency no ?

[Combuster]

(...)

I see you just met the forum looney . Don't mind him, as you've rightly guessed his OS is a mess after all.

[rdos]

for me any library or code portion that is supposed to be used by several application can fit conceptually into the os as system libraries, even if an os can also include third party libraries to provide some framework to application, it's still supposed to be part of the os bundle, if many applications need to include the same library of piece of code in order to run, it could mean the os is missing something, anything that can ease up application developpement, or allow to add easily some functionality into application can fit into the system, even if it's not directly related to kernel space

By placing user-level interface code in system libraries (like the system DLLs in Windows), you effectively make sure that kernel code (like device-drivers) cannot use this interface. That's a bad design.

kernel code shouldn't have the need to use system library, system library i think like font mannagement, audio/video/image coding/filtering, some file format manngement like compression or other like xml or what's not, but if the code has to be used by kernel device drivers yes it shouldn't be placed in user space, but anyway it's mostly likely those system library would depend on the kernel directly or indirectly so having the kernel code depending on them would make a cyclic dependency no ?

I know a particular printer that uses XML to report status information. Should we duplicate the xml-code in the device-driver? I also know a few device-drivers that needs TCP/IP, and if the socket layer is in user-space, you won't be able to use it from those drivers, rather need to duplicate it somehow in kernel space.

[h0bby1]

kernel code shouldn't have the need to use system library, system library i think like font mannagement, audio/video/image coding/filtering, some file format manngement like compression or other like xml or what's not, but if the code has to be used by kernel device drivers yes it shouldn't be placed in user space, but anyway it's mostly likely those system library would depend on the kernel directly or indirectly so having the kernel code depending on them would make a cyclic dependency no ?

I know a particular printer that uses XML to report status information. Should we duplicate the xml-code in the device-driver? I also know a few device-drivers that needs TCP/IP, and if the socket layer is in user-space, you won't be able to use it from those drivers, rather need to duplicate it somehow in kernel space.

according to me, device drivers should be self reliant, and if it need to use specific library yes it should include it, even if it need to duplicate other code in higher level api, i already see that happening in many application with the zlib who is often included and linked statically in many library, or if there is really a big need for it in several low level drivers it might be worthy to implement it in base kernel space, but if there can be issue with versions, or some particular uncommon feature to be supported by the library, drivers should link with it statically, because in term of design, device drivers are not supposed to be dependant on library being installed or not in the system

in linux they have the system to build all kind of thing directly statically in the kernel as 'built in', it can make kernel of several Mb, but it can solve the issue in some cases

socket are supposed to be part of the libc, and libc depend on the kernel, so i don't see how it could work out to have device drivers that depend on the libc who also depend on the kernel, if it only need a tcp stack for some reason, it can and should be included in the kernel, but socket are higher level

in my case the libc is included directly in the kernel and loaded very early, but i don't plan to make a socket api in it, i'd more do things in the line of the winsock2 api that i find much better than the socket system of linux, or even doing my own whole system of handling network/port/connections without socket at all, but something close to winsock2 api with event mannagement, easy asynchronous transfer mannagement without polling/select etc, but i think in any case i'll do another version of the libc to be loaded in application level and most likely file and socket mannagement will not be part of it

[rdos]

according to me, device drivers should be self reliant, and if it need to use specific library yes it should include it, even if it need to duplicate other code in higher level api, i already see that happening in many application with the zlib who is often included and linked statically in many library, or if there is really a big need for it in several low level drivers it might be worthy to implement it in base kernel space, but if there can be issue with versions, or some particular uncommon feature to be supported by the library, drivers should link with it statically, because in term of design, device drivers are not supposed to be dependant on library being installed or not in the system

How could they be self-reliant? Any meaningful device driver will at least need memory-management support and some IO-support, and you cannot place that in a static library and link it to the device driver. At the minimum, any meaningful device-driver will need to access some kernel-APIs that do memory management. And when this is needed, why not let it access the user-level API as well so you don't have to reinvent things that some devices might need like file-IO?

in linux they have the system to build all kind of thing directly statically in the kernel as 'built in', it can make kernel of several Mb, but it can solve the issue in some cases

Linux uses a lousy device-driver model where the kernel is linked to all the things that might need to be in the kernel. A much better approach is to let each device be a separate, loadable, module that uses APIs to communicate with other parts of the kernel and other device-drivers.

[VolTeK]

A much better approach is to let each device be a separate, loadable, module that uses APIs to communicate with other parts of the kernel and other device-drivers.

+1

[h0bby1]

according to me, device drivers should be self reliant, and if it need to use specific library yes it should include it, even if it need to duplicate other code in higher level api, i already see that happening in many application with the zlib who is often included and linked statically in many library, or if there is really a big need for it in several low level drivers it might be worthy to implement it in base kernel space, but if there can be issue with versions, or some particular uncommon feature to be supported by the library, drivers should link with it statically, because in term of design, device drivers are not supposed to be dependant on library being installed or not in the system

How could they be self-reliant? Any meaningful device driver will at least need memory-management support and some IO-support, and you cannot place that in a static library and link it to the device driver. At the minimum, any meaningful device-driver will need to access some kernel-APIs that do memory management.

the kernel must have a way to manage memory already anyway, kernel often have functions similar to the one of the libc to handle text output, memory mapping, i/o and other things, that are then used by the libC to provide the standard C api to higher level application, the kernel doesn't use function of the libc like malloc/printf, on window they are the function VirtualAlloc/Heapalloc, there is probably an equivalent under linux, and it's same with files, you can have memory and file mannagment in an application without the need for the libc or any library loaded, using the function of the kernel directly, but they are system specific and not portable, it's what the libc is made for

And when this is needed, why not let it access the user-level API as well so you don't have to reinvent things that some devices might need like file-IO?

normally it's more the other way around, the libC provide the standard API to access kernel function, and user level api use the libc, so you can't have the drivers depending on a lib which in turn potentially depend on him to run, as normally system lib are supposed to depend on the libc, who depend on the kernel and drivers

for drivers or thing that run in kernel space, they should use the kernel specific functions, or link anything they need that is not in the kernel statically, and those lib functions have to ebe programmed using kernel specific function for file i/o, memory, output, or anything else, because system library are not supposed to be avaible at the time of drivers load

in my case, the libc is like if it was part of the kernel, it's loaded at the very first, and then just use some kernel function for very basic things like printing, io, and base pci functions, bios calls, but it would have to be seen as a 'kernel space libc', i'd probably make another one to be used as the true libc for applications, the kernel is not linked to the libc, but all the drivers and further module are linked to it, to be able to program most of the os and base kernel function in a C/like fashion, still using kernel specific function directly for memory allocation ,it doesn't use malloc/calloc but internal system with memory zone reference with reference counter, like that there is no pointer ownsership anywhere in the kernel, and also kernel specific function for printing (no stdio ), and file system access, but most of the kernel is otherwise coded with a C-like api for memory cpy/cmp/set and string operation , and basic math things, but maybe it could be seen as a klibc, because it's libc that is just used by kernel space thing, and not a true libC meant for ansi C programming, under linu there have a whole kernel space api with kprint, kalloc or such

[rdos]

The idea was not to use the same libc in kernel and user-space, but rather to define a self-contained (doesn't depend on user level libraries) user-level API that can be used both in kernel and from user space. This basic API is then used to build libc for applications, and then the same API (+ some kernel functions) is used to build the kernel libc. Then things like malloc can be used in both applications and in kernel using the same name, but would refer to different APIs because kernel memory must be global while user memory is per-process.

[h0bby1]

it's a bit what i do for the moment, the libc in kernel space can be linked against from all applications, but the code for the functions have to be used as if they were from a dynamically linked library, if they are kernel 'built in' functions, they are not supposed to be linked against the kernel object but against the libc, so the function must be in an module external to the kernel, and the kernel can load some library in kernel space that can be linked against, but then the code for at least loading dynamically linked modules must be done without using any function of the libc, and the kernel must be able to initialize the libc as dynamically linked library without the help of any function that depend on it (which is mostly all other modules), for the moment there is only the libc, zlib, and a base library with some utilities to manipulate base system objects that are in kernel space, and any module that is loaded in the system can import the function exported by them, but i'll probably make another version of the libc to be linked against by application that will be loaded in application space without kernel access at all, the code will be duplicated for each application, but it's not very heavy anyway, and the application libc will be initialized with the memory space of the application to handle the allocations in the proper space, file i/o and sockets will be handled with functions outside of the libc, i don't plane to implement any stdio at all, i sort of hate stdio a bit i think lol the whole printf thing make it very easy to make exploit by sending chain, and the (...) elipse format is not very 'secure' all together, and i'd rather implement a better interface to handle file systems and network connections