OS Graphics

[Brendan]

Hi,

For an example, (a long long time ago) I looked at the way/s IPC is done in *nix (pipes and sockets and streams and signals and ... yuck). I didn't like it. I looked at lots of different types of messaging, and it looked a lot cleaner to me. I knew I wanted a distributed system and that synchronous messaging wouldn't work too well for normal applications, but I thought it'd be a good idea to use synchronous messaging to reduce overhead for device drivers. I implemented a "sort of monolithic" kernel where device drivers used segmentation and got loaded into the kernel space and communicated with synchronous messaging, and normal applications used asynchronous messaging to talk to each other (and to talk to device drivers). This worked fine, but I didn't like it. Then I tried the micro-kernel approach, with device drivers as processes that use asynchronous messaging. I implemented the kernel, then implemented some drivers, then implemented some applications, etc. I liked it a lot (and it's part of my OS design that I've kept ever since). Did you do something like this for RDOS? I assume you just implemented whatever seemed like a good idea in your first kernel and got stuck with it.

Of course.

How many different types of IPC did you try (and how many different kernels did you write to try them), before you found something that you thought couldn't be improved?

RDOS has a network transparent messaging API that doesn't run on top of TCP/IP. It uses it's own protocol under IP (SMP/IP), which is registered with IANA. The specification is available on my site: http://www.rdos.net/smp/ . Now please tell me that you are not distributing your system with TCP/IP, but with some efficient message protocol that also is registered with IANA?

Um, that's one. Can I assume it was the first attempt, and it seemed to work sort of, so you rushed out and registered it with IANA and never attempted any alternatives?

For an example, I had ideas for 2D image file formats rolling around in the back of my head for about 6 months. I came up with several possibilities, found problems with most of them, refined others, etc. One of those possibilities was the idea of using solid and shaded triangles for everything (as they're extremely easy to scale, rotate, render with anti-aliasing, etc). I spent about 3 months writing a prototype/utility to convert bitmap files to my file format (and convert my file format into bitmap files) and experimenting with the idea. The end result was that I discarded the "solid and shaded triangles for everything" plan - converting raw pixel data back into the least number of triangles (while getting good image quality and file sizes) proved too computationally expensive and relatively complex. It wasn't a total loss though - I've formed a new "solid, shaded and textured triangles for everything" plan, where bitmap data doesn't need to be converted but you still get the (scaling, rotating, etc) advantages. I haven't written a formal specification for the file format or implemented a prototype for it yet; but it's not my highest priority at the moment and I will one day. Did you do something like this for RDOS? I assume you just grabbed some open source PNG library and slapped it into your OS.

The graphic API was constructed from scratch. It was modeled after PicoGUI, but uses no code from that project. The GUI was also written from scratch, as was the widget toolkit. The PNG library naturally was ported as there is no sense in writing that one your own. The font interface was ported from FreeType, as there is no need to reinvent the TrueType interface either.

So you "attempted" one graphics API; where the only research you did was a google search that led you to someone else's API?

Did you try 5 different ways of porting FreeType (that all worked) before you found the perfect way to port FreeType?

For an example; so far I've probably had at least 6 different designs for "boot code" (different stages, different responsibilities, different capabilities, different ways for the pieces to communicate, etc); starting from extremely simple "text mode only" with little more than "load kernel and jump" from the early days; leading up to glorious modular pieces of art with full graphical "boot menu" windowing systems (with layers with lighting/shadows, and resolution/colour depth independence, and dithering, and fonts that are scaled to suit the physical size of the monitor), and fault tolerance features (like RAM testing and faulty RAM avoidance), and built-in CPU feature detection and "AP CPU startup", and code to auto-detect the best kernel out of many kernels to actually start. Did you do something like this for RDOS? I assume you just cobbled together a boot loader when you were just starting and haven't been able to make any significant changes to it since.

To begin with, certainly. However, there are now boot-loader for both BIOS and GRUB (multiboot). There will probably be for UEFI at some point. The job of the boot-loader is to bring up the system, and so needs nothing more than the absolute minimum in order to work.

Wow, you actually tried 2 things! Congratulations!

In regards to many kernels I feel that is an extremely bad idea. Some functions in the scheduler are different between SMP and single-core, but those are patched in the code at startup so there is no need for multiple schedulers. The kernel also detects things like the availability of PAE paging and then can install either hooks for PAE paging or normal paging. This doesn't require compiler switches or different kernels. Long mode support is loaded as an device-driver. RDOS today could run everything from DOS to 16-bit protected mode (NE), PE (Win32 and native), ELF and long mode applications without any need to configure the kernel or build multiple versions of it.

So to find the perfect kernel, you tried "lots and lots of different kernels" (one kernel) and decided it was "adequate".

Do I need to talk about memory management, or schedulers?

Those have a history of multiple versions. The one's in use today are what I consider adequate.

Just adequate?

Executable file formats?

As of above, I support many, with the same kernel.

You support many file formats (but didn't spend any time researching the design of the best file format for anything, and just used file formats designed by others)?

How about programming languages (I'm currently researching that - designing languages and writing a crappy tool-chain)?

BCC, OpenWatcom, GCC.

How much time did you spend researching languages before you invented C; and how long did it take you to implement these compilers?

To be honest; I don't think anyone here (other than me) has done anything like this for their project.

You're wrong.

No, your idea of "researching many alternatives" is to settle for the first thing that comes along. This makes sense for your project, where time is limited and you can't afford to (e.g.) throw out several working kernels in a search for perfection (mostly because there are existing customers that depend on the old code).

What I can't understand is why you're attempting to lie about it.


Cheers,

Brendan

[Brendan]

Hi,

To be honest; I don't think anyone here (other than me) has done anything like this for their project.

After four kernel rewrites, I'd beg to differ.

Don't misunderstand me - I know there are a lot of people that do kernel rewrites, etc.

What I'm saying is that (as far as I know) nobody else is trying to find the perfect design for everything (including kernel, programming languages and tools, all file formats, networking protocols, etc). Most people are much more practical, and will (e.g.) decide to use GCC, or PNG, or HTML instead of attempting to change the world.

Note: I'm not actually trying to change the world. What I'm trying to do is create an alternative world that is separate from (and untainted by) the world we have now.


Cheers,

Brendan

[rdos]

Um, that's one. Can I assume it was the first attempt, and it seemed to work sort of, so you rushed out and registered it with IANA and never attempted any alternatives?

I researched the alternatives, which all operated on top of TCP/IP and decided it was not any good (because TCP is stream-oriented, not message oriented and UDP has no support for large messages and transmission control). Then I designed a superior alternative myself and then implemented it. Since I haven't come up with any reason why I would redesign it, I haven't done any more on it.

Did you try 5 different ways of porting FreeType (that all worked) before you found the perfect way to port FreeType?

At least two. The first was done in user space, but that was not considered acceptable since I didn't want to limit font rendering to only user-space. The second attempt was as an device-driver.

To begin with, certainly. However, there are now boot-loader for both BIOS and GRUB (multiboot). There will probably be for UEFI at some point. The job of the boot-loader is to bring up the system, and so needs nothing more than the absolute minimum in order to work.

Wow, you actually tried 2 things! Congratulations!

Not tried. I implemented the boot process in two different ways. The multiboot implementation was motivated by a need to dual-boot with Windows.

So to find the perfect kernel, you tried "lots and lots of different kernels" (one kernel) and decided it was "adequate".

I don't think we put the same meaning into a "kernel". In my OS, the kernel is basically only interrupt handling, memory handling and a device-driver loader. Thus, there is no need for multiple kernels as those things can only be done in a limited number of ways, and when there are alternatives (like paging), one could implement all alternatives and select the most adequate in the boot process. Not even the scheduler is part of the kernel and could at least theoretically be replaced by something else without touching the kernel.

You support many file formats (but didn't spend any time researching the design of the best file format for anything, and just used file formats designed by others)?

Is there any sane reason for creating your own executable format? I don't think so.

How much time did you spend researching languages before you invented C; and how long did it take you to implement these compilers?

Is there any sane reason why to write a compiler for an OS as opposed to port a C library or similar? I don't think so.

No, your idea of "researching many alternatives" is to settle for the first thing that comes along. This makes sense for your project, where time is limited and you can't afford to (e.g.) throw out several working kernels in a search for perfection (mostly because there are existing customers that depend on the old code).

The issue of depending on already running installations is relatively new. For most of the time, there has been no problems with changing anything other than to make it work.

And no, I'm not a perfectionist. I believe that working systems are more attractive than perfect systems, as the perfect system seldom is working properly (if it runs at all).

[rdos]

What I'm saying is that (as far as I know) nobody else is trying to find the perfect design for everything (including kernel, programming languages and tools, all file formats, networking protocols, etc). Most people are much more practical, and will (e.g.) decide to use GCC, or PNG, or HTML instead of attempting to change the world.

You can only change the world if you are realistic about your ambitions. If you have a perfectionist view where you want to change almost everything in OS development, and also change your designs as soon as you find something that is even more perfect, you are not likely to be able to change anything significant.

[dozniak]

Note: I'm not actually trying to change the world. What I'm trying to do is create an alternative world that is separate from (and untainted by) the world we have now.

Brendan's idealism is so shining through now, I'm ready to side with rdos.
(Edit: But I stand by myself instead.)

[Brendan]

Hi,

You support many file formats (but didn't spend any time researching the design of the best file format for anything, and just used file formats designed by others)?

Is there any sane reason for creating your own executable format? I don't think so.

The only reason I need is that (ignoring compatibility concerns) I can think of ways to make the file format/s better for my OS.

Note: I'm not just talking about executable file formats though - I mean *all* file formats. For example, I'm not supporting "plain text", or PNG, or HTML, or MPEG, or PDF, or .... I'll do my own research and create my own alternative file formats for everything.

How much time did you spend researching languages before you invented C; and how long did it take you to implement these compilers?

Is there any sane reason why to write a compiler for an OS as opposed to port a C library or similar? I don't think so.

Here's 3 reasons:

• Ignoring compatibility concerns; I can think of ways to improve the programming language
• Ignoring compatibility concerns; I can think of ways to improve the input file format
• Ignoring compatibility concerns; I can think of ways to improve the output file format

And no, I'm not a perfectionist. I believe that working systems are more attractive than perfect systems, as the perfect system seldom is working properly (if it runs at all).

If I were happy with working systems (including my own previous working systems) I wouldn't be writing an OS.


Cheers,

Brendan

[rdos]

Note: I'm not just talking about executable file formats though - I mean *all* file formats. For example, I'm not supporting "plain text", or PNG, or HTML, or MPEG, or PDF, or .... I'll do my own research and create my own alternative file formats for everything.

I think such things are more attractive to try to improve on in relation to specific projects. For instance, it is possible to create better image formats, but such thing is better achieved when done on a main-stream OS than on a hobby-OS. Same goes for HTML, MPEG and PDF. You will have greater success by providing better standards that work on some or all mainstream OSes than if you also roll your own OS project.

And AFAIK, there is no need to combine such new designs with also designing an OS. That's kind of like writing your own OS when you only want to write some specific application. Could be fun, but it is hardly rational.

Here's 3 reasons:

• Ignoring compatibility concerns; I can think of ways to improve the programming language
• Ignoring compatibility concerns; I can think of ways to improve the input file format
• Ignoring compatibility concerns; I can think of ways to improve the output file format

Sure, there could be done much on compilers. The question is what needs to be done on compilers that has no relevance for mainstream OSes. I think not a lot.

If I were happy with working systems (including my own previous working systems) I wouldn't be writing an OS.

But MS-DOS wasn't a working system for multitasking, so I had a rational reason to write a new OS that could work with multitasking.

And when M$ finally launched something with decent multitasking, I already had a stable product and a lot of new features that M$ didn't support. So at that point the choice to continue with OS development was still a rational one.

[Griwes]

Note: I'm not actually trying to change the world. What I'm trying to do is create an alternative world that is separate from (and untainted by) the world we have now.

Brendan's idealism is so shining through now, I'm ready to side with rdos.
(Edit: But I stand by myself instead.)

This.

Note: I'm not just talking about executable file formats though - I mean *all* file formats. For example, I'm not supporting "plain text", or PNG, or HTML, or MPEG, or PDF, or .... I'll do my own research and create my own alternative file formats for everything.

...which is just dumb. Putting aside the fact that PNG or MPEG or PDF might be severely broken (but as they are not the area where I spent much time on researching things, I don't know if they are), but they (surprise!) work quite well for current system.

And reinventing "plain text" is just plain *stupid*. It's *plain text*, goddammit, what else do you want it to be? Hundreds of KiB of totally unwanted metadata, when the end user wants just *plain old plain text file*?

Not supporting HTML is even dumber, for very obvious reasons.

Disclaimer: I do remember that you once said that the only end user you care about is you, but if you don't care about other potential end users, you basically end up being another rdos who always says "for my specific case it works, so it must be the best solution". Tip: it's a logical fallacy. And while I will, for example, use my own executable file format for my OS, I will still *support* the old ones. Making people use your "alternative" when the old ones are not available is nowhere near creating anything superior; you know you've created something superior, when people decide to use your thing over others, when all the options are freely available for them to choose from.

[Brendan]

Hi,

What I'm saying is that (as far as I know) nobody else is trying to find the perfect design for everything (including kernel, programming languages and tools, all file formats, networking protocols, etc). Most people are much more practical, and will (e.g.) decide to use GCC, or PNG, or HTML instead of attempting to change the world.

You can only change the world if you are realistic about your ambitions. If you have a perfectionist view where you want to change almost everything in OS development, and also change your designs as soon as you find something that is even more perfect, you are not likely to be able to change anything significant.

Your ambitions are very realistic. How much of the world will RDOS change in the next 20 years? Shall we expect a press release from Apple advising their stockholders that they're going to stop bothering with OS X because they've seen RDOS and were very impressed?


Cheers,

Brendan

[dozniak]

How much of the world will RDOS change in the next 20 years? Shall we expect a press release from Apple advising their stockholders that they're going to stop bothering with OS X because they've seen RDOS and were very impressed?

Are you per chance expecting the same kind of press release from Apple about your OS? Apple guys are hard at work polishing compatible stuff, like OpenCL; hundreds of them bring out things like Clang, Objective-C runtimes with ARC - all not reinvented from scratch, but built on previous foundations.

They will not be impressed with your OS either.

[rdos]

Your ambitions are very realistic. How much of the world will RDOS change in the next 20 years? Shall we expect a press release from Apple advising their stockholders that they're going to stop bothering with OS X because they've seen RDOS and were very impressed?

I imagine that RDOS will run on some thousand systems, and that our current end product might become rather well-known. I also imagine I will do support on those installations for a while, even after I have retired. That's good enough for me. That keeps me busy with programming tasks that won't require me learning anything about new features in mainstream OSes like Windows or Linux or any other IT related hypes.

I expect a lot more from my new interest that I have only pursued since about 2001.

So, in short, for the moment it seems like a nice way to earn money for a living, but I don't expect much more than that.

[Brendan]

Hi,

Note: I'm not actually trying to change the world. What I'm trying to do is create an alternative world that is separate from (and untainted by) the world we have now.

Brendan's idealism is so shining through now, I'm ready to side with rdos.
(Edit: But I stand by myself instead.)

This.

I'm not claiming this is the most practical approach..

Note: I'm not just talking about executable file formats though - I mean *all* file formats. For example, I'm not supporting "plain text", or PNG, or HTML, or MPEG, or PDF, or .... I'll do my own research and create my own alternative file formats for everything.

...which is just dumb. Putting aside the fact that PNG or MPEG or PDF might be severely broken (but as they are not the area where I spent much time on researching things, I don't know if they are), but they (surprise!) work quite well for current system.

And reinventing "plain text" is just plain *stupid*. It's *plain text*, goddammit, what else do you want it to be? Hundreds of KiB of totally unwanted metadata, when the end user wants just *plain old plain text file*?

What actually is plain text? Is it ASCII or UTF8 or something else? How wide are tabs? What are the standard "end of line" character/s?

How easy is it for software to tell the difference between a plain text file and other file formats? How easy is it for software to detect if a plain text file is corrupt?

Not supporting HTML is even dumber, for very obvious reasons.

If I wasn't trying to create my own little world; then not supporting HTML would be silly.

Maybe.

To be honest, I don't know. HTML is the start of a large and ugly pile of puke (CSS, Javascript, XML, ...) and I really do wish everyone would stop listening to W3C so that we can at least hope for something better one day.

Disclaimer: I do remember that you once said that the only end user you care about is you, but if you don't care about other potential end users, you basically end up being another rdos who always says "for my specific case it works, so it must be the best solution". Tip: it's a logical fallacy. And while I will, for example, use my own executable file format for my OS, I will still *support* the old ones. Making people use your "alternative" when the old ones are not available is nowhere near creating anything superior; you know you've created something superior, when people decide to use your thing over others, when all the options are freely available for them to choose from.

Currently there is one world with many OSs; and people choose between "smelly old world on top of Windows", "smelly old world on top of Linux", "smelly old world on top of OS X", etc. It doesn't matter which OS you choose you get "smelly old world" with it.

I'm creating a new world, so that people can choose "nice new world on top of BCOS". Hopefully someone else will write a competing OS for the new world, and people will be able to choose "nice new world on top of <other OS>" too. That'd be great (but unlikely).

Now; you're suggesting that I should write "smelly old world on top of BCOS". From my perspective, it's like someone living in a huge city that wants to escape all the concrete by building a nice green park, and then you saying "Hey, you should put some concrete buildings on that park!".


Cheers,

Brendan

[Griwes]

No, all you are trying to do is comparable more to building an own house on the Moon, instead of just in some distant place on Earth, and to pretending everything the civilization has ever achieved was a pile of puke and you alone can redo everything far better than anybody else in the history. And that you are some kind of super human capable of knowing everything.

Tip: you cannot get anywhere near being successful that way.

And plain text file is obviously UTF-8, and if you want specific spacing, you obviously just use appropriate amount of spaces instead of those silly tabs.

[rdos]

I'd follow in Brendan's trails if it wasn't because I know it's not worthwhile pursuing. IOW, I agree with his idea that the IT world has gone bankrupt by bad design and code, and that also includes most open-source projects. It is end users that cannot make a simple program, and sometimes cannot even use a simple program that determines the decisions of the software industry. In addition to that, big software companies have produced so many conflicting standards that they effectively have locked-up software programmers even if they are not closed anymore.

In essence, being a programmer on "modern" tools means you need to dedicate most of your time learning crap that big companies made in order to tie programmers up on their solutions so they have no time for inventing their own.

[XanClic]

Please understand that there's a massive difference between "realistic rendering" and "reality". For example, you can have a game where the player is a yellow sphere with a slit for a mouth who runs (without arms or legs) through corridors filled with pills that float in mid-air (un-effected by gravity) while trying to avoid ghosts; and regardless of how realistic the graphics are for this game, if you turn the computer off and go outside to "reality" you won't see yellow spherical people running around, or pills floating in mid-air, or ghosts. Trying to suggest that "realistic rendering" has anything to do with "reality" is silly.

Of course realistic rendering is linked to reality. The name is the first evidence. Second, I was talking about photorealistic rendering on purpose. The very definition of that is that you try to create an image which looks like something a camera might have taken. I'm very well aware of the fact that the process in which that image is generated on today's GPUs has very little to do with the process "invoked" by reality. However, the purpose still is to copy the results of reality, thus it is obviously linked to reality. You took my statements about "photorealistic rendering" for statements about "realistic rendering" - so you seem to imply they're both the same. I'd agree with you on that one. But if they are the same, realistic rendering is also very much linked to reality - its result, not the process used (actually, the process is undefined).
So when you're saying one can use (photo-)realistic rendering to generate non-realistic pictures, this is by definition just wrong.

However, I see where you're coming from. The processes commonly involved in photorealistic rendering (for god's sake, this is just such a long word - I'll abbreviate it as PRR from now on) can be slightly modified to generate pictures which are commonly seen as non-realistic, for instance, using toon shading. However, to believe this will then be a superset of everything you'll ever want to render is just as wrong as depicting aliens as humanoids. Some people say "Oh, he has green skin, he can't be human" although that "alien" is just a human painted green. Alien life might just be totally alien to us, but many Sci-Fi movies picture them very alike to organisms from earth. In the same way, you picture rendering to be always a special case of PRR (or, to be more exact, as a special case for a PRR capable renderer), which is a very narrow view. Someone might come up with something completely different which we can't even imagine yet.

Anyway, there is also a very common example today: 2D rendering. As you will probably know, most 2D games today are rendered on the GPU using the 3D hardware. The process they're using is utterly unrelated to PRR. But I guess you'll just implement this as well, then (although there isn't even a "common" implementation as there is for 3D PRR).

I'd just add a "load_shader" command (that tells the video card to load a shader that uses some sort of standardised byte-code for shader language, like GLSL)

First off, GLSL is no byte code (OpenGL does in fact specify a very vague format for compiled shaders, but this isn't portable between different GPUs). Second, you do know that shaders aren't just some trivial stuff which you can throw into any 3D engine and the result will always be the same? Shaders heavily depend on the engine architecture involved. The shaders would have to be written specifically for your rendering architecture - but I guess you'll just ignore this since you ignore everything which isn't written specifically for your system. Also, the interface you provide to the shaders would have to be stable, but that problem is actually somehow solvable.

Third, shaders aren't everything. Although today's rendering pipelines heavily depend on shaders, there is still a whole lot which happens outside of them. But I guess you'd then be free to specify input data to the shaders as well as textures on your system so you'd actually be able to use an API equivalent to, e.g., OpenGL (although you could just provide that directly, then, but I'll come back to that point).

Basically; the entire discussion about shaders has nothing to do with "list of commands vs. raw pixels" anyway.

Of course not - though I wasn't the one to start this off-topic branch. Frankly, I don't even care about that original topic anyway, since that question can be solved by doing some benchmarks on both approaches.

If someone was giving you a free Ferrari and told you the car is capable of doing 340 km/h, would you tell them that you don't want the free Ferrari because sometimes you want to drive slower? If someone gave you a free OS that's capable of supporting 3D GUIs well, why would you immediately assume that every GUI for that OS must use all the 3D features all the time (and fail to understand that a GUI for that OS could also be made to look as crappy as some monochrome 2D GUI from 3 decades ago)?

OK, you're right. If I can use your GUI in the plain old 2D mode, I'm fine. Although I then don't understand why I should prefer your GUI over any other, since I can hardly imagine a practical use for the additional features.
At least for the Ferrari, I could, since I'm a German capable of driving on German Autobahns (though I don't see the point; I rarely exceed 160 km/h, so a Ferrari actually has no benefit to me over a Ford) .

Rather than thinking about how different pieces of software communicate (e.g. how applications communicate with GUIs, what sort of video driver API to use, etc) and trying to make the interfaces between these pieces better (e.g. more flexible); I should just gather random pieces of code that other people have written for completely different purposes and stitch them together in strange ways (e.g. slap code from the unreal developer's kit into things like text editors, widgets, GUIs, etc), and then hope that the result won't be a hideous abomination?

Um, no? You should just do what everyone else is doing, i.e., implement a "low-level" interface such as OpenGL and then use libraries for the rest? But I guess that's no solution to you, since you hate the idea of libraries. I love it. I guess that pretty much sums up why I'm very sceptical about a whole 3D/physics engine as part of the OS and I rather prefer the ability to choose from a variety of libraries to achieve my goal (or even using the low-level interface in the first place) and why you are of such a different opinion.