Sup, new to OS development; Here's my project, NucleoOS
Posted: Sat Feb 11, 2012 12:54 pm
Sup! I'm not sure if there's even a subforum to introduce yourself, but I decided that doing it somewhere in the offtopic section would work, it's about as ontopic as "that funny thing your cat did". 
Anyways, I decided to go into OS development a few months ago, and kind took a crash course of my own on OS design. I'm planning and currently writing the hardware interfaces for my first OS, for the 84+/84+SE on the standard zilog z80 processor. I'm experienced with C, but also very experienced with z80 assembly, and I'm semi-experienced with Renesas SuperH 3/4/4-A and x86/x86-64 assembly. Here is a plan of my OS, posted elsewhere (more of a general programming forum that I reside at):
NucleoOS -- An OS for the 84+/84+SE
For months I have been catching up on my knowledge of z80 assembly, 84+ hardware, OS design, and many other things, so that I could begin to work on a project I've been considering tackling for a long time: a simple, but fully fledged operating system for the 84+ and 84+SE. My quest finally began earlier this week, when I set up a nice little dev environment for my operating system, and started organizing my concept code for the layout and how different modules would work together. I currently am still working on setting up hardware and such, not much more than a period of time spent fleshing out initializing components, and RTFMing at WikiTI (thanks for the site Brandon, and everyone else who submitted information there -- it's extraordinarily helpful).
I have a few inspirations in mind for my endeavors; namely, MenuetOS (x86) for it's simplicity and very small size, KOS (z80) for its decently handled process scheduling, and GlassOS (z80) to some extent for cranking the full possible use out of the 84+'s hardware. The name of the project is NucleoOS, since Nucleo in Italian means "core"; the main goal of NucleoOS is to be as small as possible, only holding the core necessities and a few limited "sweets" in its hull. It is planned to sport the following:
multiprocessing capabilities
Likely the largest planned element, and placed closest to my personal expectations goals for this whole project. Planned to support to 16 processes running at the same time, accomplished in a mix of Round-robin and Priority ranking scheduling techniques. Each process will be given a flash page for execution in bank $4000-$7FFF, which allows for consistency in program execution space, and in a sense a slight feeling of pseudo-virtualization of process memory. Processes taking more than 2^14 bytes of memory, and therefore not fitting into a single execution flash page, will have to act similarly to how multi-paged flash apps do on TI-OS -- manually call for swapping pages through the OS to access later chunks of code or data; single page processes will not have to worry about this at all. Processes can have a priority level of 0-3. The values corresponding to priorities are as follows, with some examples of the types of processes that would be in these priority ranks:
0 - services, very idle (link-port/battery monitor)
1 - non-interactive, background tasks (background real time clock)
2 - interactive, CPU/IO semi-intensive (text editor)
3 - real-time, CPU/IO intensive (games, something using the LCD for grayscale)
Rank 3 processes will get ~70% shared CPU time, 2 gets ~20%, 1 gets ~8%, and services get ~2%. Processes are time sliced in order in a round robin formation, but get a higher throughput ratio and longer timeslice based on their rank. Processes will have the ability to halt for a set period of time; if the time limit is reached during another process's timeslice, and the currently running process is of a lower or equal rank, execution of the current process will be forked over to the other process of which the limit was reached; if the current process is more critically ranked, execution will be forked over to the limit-reaching process after the current process has finished its slice.
Threading within a process so that both may share the same execution space and CPU time is semi-planned out at this point; I'm thinking of having a system call to initialize new threads, with given label/addresses within the current process:
thread_init: executed on creation of thread, ran to entirety, ignoring thread/process scheduling
thread_run: executed on a thread's timeslice within the process timeslice
thread_kill: executed upon the killing of the thread, and this whole set of code will be run to it's entirety, ignoring thread/process scheduling
Scheduling will be handled through hardware timer-based interrupts set at a higher speed, in IM 1. I plan on having a Short-term/Dispatcher pair working together to shift all the processes and their execution flash page in and out of the bank; the short term scheduler keeps track of how many scheduler time slice calls from the interrupt a process has run through, to handle priority ranking effectively. A long term scheduler is planned to keep the amount of running processes relatively clean, so that very idle processes in ranks 1-3 will be set aside until their wanted wakeup signal is reached (essentially, a signal that tells that the amount of high-ranked processes has dropped quite a bit, and there is more CPU time for them to take advantage of); services take up very little time as it is, so they won't be affected by the long term scheduler. The long term scheduler is planned to be very minimal, only being activated every 16 or so short-term runs, to minimize slowdowns where possible. Since the programs aren't all stored in the same chunk of memory at the same time, and therefore don't need to be swapped out per-se, there is no mid term scheduler planned.
Primitive MMU (Memory Management)
Since the 84+(SE) has code execution limit hardware, I am putting it to use so that executable code running only in user space in bank $4000-$7FFF is limited to running almost strictly in that address space. If a jump is made outside that realm, a reset will occur, which NucleoOS will catch and determine if the reset was caused by a process, and if so will call for the process to be terminated and set the last recorded signal to be SIGMEMACC (signal message illegal memory access). The definition between user space and priv space is solely determined by address range, and when execution veers out of user space back into priv space, I.e. With the short term scheduler interrupt, the limits are relieved and the OS has execution space of $0000-$7FFF.
The hardware enabling this is fortunately there, but unfortunately, it is extremely limited, almost solely to page and address execution limits; therefore, it would be possible for user space programs to take control of priv space, but since the whole point of memory management and priv leveling here is to make user space programs generally safer, there's really no incentive of the user hacking into the internals, since I'll likely provide some syscalls to do some of that with services and rank 1 processes anyways. I could always scan a program to see if it edits the limits of these ports before dispatching it, but what's the fun in making an OS like the restrictive one made by TI corporate pigs? The whole point is safety and organization, but if the user wants to go amuck with the system, they're open to do it, it's their calculator, not mine
Simple device drivers/hardware interaction modules
I plan on making hardware interfacing routines in driver modules for some devices up front (some of which are already coded):
LCD - provide an easy way to send/get LCD data, change some status modes, very simple sprite routines, and allow for use of the LCD's internal RAM for extra scratch space (thinking of options on how to do this, and if I'll only allow it if no other process is using it)
Link port - not thought out much; most driver functions will likely be for sending some simple data, setting the line states, etc.
USB - far out goal, near the end of my actual priorities.
Timers - support for using the second and third crystal timers easily.
Coding Timeline
This project is now finally on my list of project in the works, and will stay that way; it'll likely be lower on scale than TaN, replacing most of my time being spent elsewhere on Raptor. Don't expect super fast progress. I hope to have basic CPU control, the LCD driver, and a few simple OS routines done in a week or so, and the goal after that will be to get full multiprocessing done by sometime in summer. I feel ready with the knowledge base to take on such a project, and I am 100% aware of the amount of time it will require to fulfill even the basic operations. I don't even have no plan of a GUI yet, or for that matter, even some sort of terminal emulator, so I'm hopefully not off on the wrong track of expectations that others have run into
I will be asking help in this thread as often as I will be updating; until a few months ago when I started my OS inner workings crash course, I knew nothing about how an OS should work, and I'm still far from knowing a lot. Most questions will likely be technical ones about why some code is causing nasty stuff to happen, but I'll definitely be asking for help with advice for general OS structure.
I'm not very far into actual coding now, so if you see something in my plans above that sounds awful or plain wrong, please tell me if you can! My only reason for taking on this project is as a personal learning experience, so I'm fine with blunt answers or advice; I'll learn more that way -- hit me with your best design rebuttals!
(PS -- sorry for being so technical with my descriptions above, I want to throw as much out here as I can for the best advice, and to show I actual put thought/research into all this; this is also all the current digital documentation available right now, most other documentation is drawn out diagrams of how things would work together)
In addition, when queried with some questions, I answered with some extra technical questions:
basically, the first page of RAM is planned to use a small stack, virtual BP and SP "registers", and open space for processes (for now, probably 800 bytes). The second page is basically purely for dynamically allocated memory, and the third will hold some extra space for processes that can be requested, system flags, the kernel stack, executable code space for doing hardware-related stuff that needs to be done in RAM (probably the dispatch sheduler) and a few other things. On models with extra RAM, I'm unsure what to do with it, though it'd probably be requestable RAM or even something like swap space of some sort.
Basically, here's all the plans for my OS Kernel; any suggestions based on the above? Anything sound stupid, or bright? Thanks in advance for any tips
Anyways, I decided to go into OS development a few months ago, and kind took a crash course of my own on OS design. I'm planning and currently writing the hardware interfaces for my first OS, for the 84+/84+SE on the standard zilog z80 processor. I'm experienced with C, but also very experienced with z80 assembly, and I'm semi-experienced with Renesas SuperH 3/4/4-A and x86/x86-64 assembly. Here is a plan of my OS, posted elsewhere (more of a general programming forum that I reside at):
NucleoOS -- An OS for the 84+/84+SE
For months I have been catching up on my knowledge of z80 assembly, 84+ hardware, OS design, and many other things, so that I could begin to work on a project I've been considering tackling for a long time: a simple, but fully fledged operating system for the 84+ and 84+SE. My quest finally began earlier this week, when I set up a nice little dev environment for my operating system, and started organizing my concept code for the layout and how different modules would work together. I currently am still working on setting up hardware and such, not much more than a period of time spent fleshing out initializing components, and RTFMing at WikiTI (thanks for the site Brandon, and everyone else who submitted information there -- it's extraordinarily helpful).
I have a few inspirations in mind for my endeavors; namely, MenuetOS (x86) for it's simplicity and very small size, KOS (z80) for its decently handled process scheduling, and GlassOS (z80) to some extent for cranking the full possible use out of the 84+'s hardware. The name of the project is NucleoOS, since Nucleo in Italian means "core"; the main goal of NucleoOS is to be as small as possible, only holding the core necessities and a few limited "sweets" in its hull. It is planned to sport the following:
multiprocessing capabilities
Likely the largest planned element, and placed closest to my personal expectations goals for this whole project. Planned to support to 16 processes running at the same time, accomplished in a mix of Round-robin and Priority ranking scheduling techniques. Each process will be given a flash page for execution in bank $4000-$7FFF, which allows for consistency in program execution space, and in a sense a slight feeling of pseudo-virtualization of process memory. Processes taking more than 2^14 bytes of memory, and therefore not fitting into a single execution flash page, will have to act similarly to how multi-paged flash apps do on TI-OS -- manually call for swapping pages through the OS to access later chunks of code or data; single page processes will not have to worry about this at all. Processes can have a priority level of 0-3. The values corresponding to priorities are as follows, with some examples of the types of processes that would be in these priority ranks:
0 - services, very idle (link-port/battery monitor)
1 - non-interactive, background tasks (background real time clock)
2 - interactive, CPU/IO semi-intensive (text editor)
3 - real-time, CPU/IO intensive (games, something using the LCD for grayscale)
Rank 3 processes will get ~70% shared CPU time, 2 gets ~20%, 1 gets ~8%, and services get ~2%. Processes are time sliced in order in a round robin formation, but get a higher throughput ratio and longer timeslice based on their rank. Processes will have the ability to halt for a set period of time; if the time limit is reached during another process's timeslice, and the currently running process is of a lower or equal rank, execution of the current process will be forked over to the other process of which the limit was reached; if the current process is more critically ranked, execution will be forked over to the limit-reaching process after the current process has finished its slice.
Threading within a process so that both may share the same execution space and CPU time is semi-planned out at this point; I'm thinking of having a system call to initialize new threads, with given label/addresses within the current process:
thread_init: executed on creation of thread, ran to entirety, ignoring thread/process scheduling
thread_run: executed on a thread's timeslice within the process timeslice
thread_kill: executed upon the killing of the thread, and this whole set of code will be run to it's entirety, ignoring thread/process scheduling
Scheduling will be handled through hardware timer-based interrupts set at a higher speed, in IM 1. I plan on having a Short-term/Dispatcher pair working together to shift all the processes and their execution flash page in and out of the bank; the short term scheduler keeps track of how many scheduler time slice calls from the interrupt a process has run through, to handle priority ranking effectively. A long term scheduler is planned to keep the amount of running processes relatively clean, so that very idle processes in ranks 1-3 will be set aside until their wanted wakeup signal is reached (essentially, a signal that tells that the amount of high-ranked processes has dropped quite a bit, and there is more CPU time for them to take advantage of); services take up very little time as it is, so they won't be affected by the long term scheduler. The long term scheduler is planned to be very minimal, only being activated every 16 or so short-term runs, to minimize slowdowns where possible. Since the programs aren't all stored in the same chunk of memory at the same time, and therefore don't need to be swapped out per-se, there is no mid term scheduler planned.
Primitive MMU (Memory Management)
Since the 84+(SE) has code execution limit hardware, I am putting it to use so that executable code running only in user space in bank $4000-$7FFF is limited to running almost strictly in that address space. If a jump is made outside that realm, a reset will occur, which NucleoOS will catch and determine if the reset was caused by a process, and if so will call for the process to be terminated and set the last recorded signal to be SIGMEMACC (signal message illegal memory access). The definition between user space and priv space is solely determined by address range, and when execution veers out of user space back into priv space, I.e. With the short term scheduler interrupt, the limits are relieved and the OS has execution space of $0000-$7FFF.
The hardware enabling this is fortunately there, but unfortunately, it is extremely limited, almost solely to page and address execution limits; therefore, it would be possible for user space programs to take control of priv space, but since the whole point of memory management and priv leveling here is to make user space programs generally safer, there's really no incentive of the user hacking into the internals, since I'll likely provide some syscalls to do some of that with services and rank 1 processes anyways. I could always scan a program to see if it edits the limits of these ports before dispatching it, but what's the fun in making an OS like the restrictive one made by TI corporate pigs? The whole point is safety and organization, but if the user wants to go amuck with the system, they're open to do it, it's their calculator, not mine
Simple device drivers/hardware interaction modules
I plan on making hardware interfacing routines in driver modules for some devices up front (some of which are already coded):
LCD - provide an easy way to send/get LCD data, change some status modes, very simple sprite routines, and allow for use of the LCD's internal RAM for extra scratch space (thinking of options on how to do this, and if I'll only allow it if no other process is using it)
Link port - not thought out much; most driver functions will likely be for sending some simple data, setting the line states, etc.
USB - far out goal, near the end of my actual priorities.
Timers - support for using the second and third crystal timers easily.
Coding Timeline
This project is now finally on my list of project in the works, and will stay that way; it'll likely be lower on scale than TaN, replacing most of my time being spent elsewhere on Raptor. Don't expect super fast progress. I hope to have basic CPU control, the LCD driver, and a few simple OS routines done in a week or so, and the goal after that will be to get full multiprocessing done by sometime in summer. I feel ready with the knowledge base to take on such a project, and I am 100% aware of the amount of time it will require to fulfill even the basic operations. I don't even have no plan of a GUI yet, or for that matter, even some sort of terminal emulator, so I'm hopefully not off on the wrong track of expectations that others have run into
I will be asking help in this thread as often as I will be updating; until a few months ago when I started my OS inner workings crash course, I knew nothing about how an OS should work, and I'm still far from knowing a lot. Most questions will likely be technical ones about why some code is causing nasty stuff to happen, but I'll definitely be asking for help with advice for general OS structure.
I'm not very far into actual coding now, so if you see something in my plans above that sounds awful or plain wrong, please tell me if you can! My only reason for taking on this project is as a personal learning experience, so I'm fine with blunt answers or advice; I'll learn more that way -- hit me with your best design rebuttals!
(PS -- sorry for being so technical with my descriptions above, I want to throw as much out here as I can for the best advice, and to show I actual put thought/research into all this; this is also all the current digital documentation available right now, most other documentation is drawn out diagrams of how things would work together)
In addition, when queried with some questions, I answered with some extra technical questions:
Good point -- I'm still in early design of the filesystem, so I'm drawing out how it might work. It would be cool to use a widely-used ext filesystem, that would probably be either too large or slow to work with; what type of system does GlassOS use?AHelper wrote:Happy to hear of another OS starting up!
An important thing to think about is he flash chip and how it will be used. A filesystem (or other storage method is needed in order to run programs on the calc. Second thing to think about is how to put programs on the calc. Linkport is great if you have the ability to use it (which is why GlaßOS has USB and no IO code). This will only be needed once the kernel is mature and the functionality is ready for processes (you can embed test programs or code into the 8xu).
Not a bad bit of input -- though I'm still sure some apps would have to have over 2^14 memory space available, so I might still have to go this route.Second point, I see that you mentioned that multi-paged app functionality will be allowed. I will throw out the idea of shared libraries.
So each process has it's own LCD buffer assigned already at dispatch? Hmm, not a bad idea, actually.For the LCD, be careful when allowing the user to background tasks that use their own drawing. If you have one buffer and the program isn't made aware that the lcd is dirty, then you would have a process resumed and the lcd would not be updated. GlaßOS has an lcd buffer for each process and restores it when resuming. If you want only one, then I suggest some sort of signal sending or flag setting method to alert about a dirty lcd buffer.
Me and Fishbot talked about this last night, and basically I'm coming to the conclusion that this part of the whole thing might be chucked. My original plan was to catch it by having some sort of flag in flash memory that would update with each process switch, but that would probably slow down the short term scheduler now that I think of it, and really wear down the flash chip. If anyone familiar with how to control a reset caused by ports 25h & 26h can help with conceptualizing, that would be superb!For the memory execution protection hardware, someone correct me if needed, but this simply reboots the calc. You will not be able to catch it as having the calc reset from a battery pull starts the os the same way an execution reset starts it.
I'm thinking of somehow reserving the first crystal timer for this, but there's not currently as much priority for this right now so I'll leave that up to me later on to figure out.For priority 3 processes that use greyscale, would the greyscale drawing be done? Manually via a timer? OS-provided?
Oh crap, I forgot to talk about RAM usageAlso, in the current likely state of the 84pse, how will you divide up 48 KB of ram into 32 pieces, a stack and other data section per each 16 processes?
basically, the first page of RAM is planned to use a small stack, virtual BP and SP "registers", and open space for processes (for now, probably 800 bytes). The second page is basically purely for dynamically allocated memory, and the third will hold some extra space for processes that can be requested, system flags, the kernel stack, executable code space for doing hardware-related stuff that needs to be done in RAM (probably the dispatch sheduler) and a few other things. On models with extra RAM, I'm unsure what to do with it, though it'd probably be requestable RAM or even something like swap space of some sort. Thanks, I'll definitely need it!Good luck!
Basically, here's all the plans for my OS Kernel; any suggestions based on the above? Anything sound stupid, or bright? Thanks in advance for any tips
