Reinventing Unix is not my problem

[spotracite]

After looking through these forums and the Wiki, I've come to the conclusion that "Unix" is a very divisive term among this community. On Side A, we have the Unix haters (myself included) who have decided that, for some reason or another, Unix is an architectural dead-end and should be replaced with more modern technology (most often, Windows NT). On Side B, we have the Unix lovers (again, no offense) who have decided that, for some reason or another, Unix is the pinnacle of operating system architecture and should either be expanded on by 'X' feature or is already perfect. Personally, I believe that Unix isn't great, but isn't terrible - some design choices are very weird, some are downright bad, but in general I think that Unix is a good middle-of-the-road system (especially for something written in just a few months).

A specific quote, however, from 1987 on the Usenet, draws my attention: "Those who do not understand Unix are condemned to reinvent it, poorly."

This quote brings up a major issue (in my opinion) about the OSDev community - Most hobby operating systems lean towards the Unix-like variety. Why is this? I think it can be argued that it's easy to understand, the simplicity makes its implementation easy, or perhaps POSIX is just appealing because of how well defined it is. However, I have a different argument for this: Hobby operating systems, especially beginner operating systems, skew towards poorly reinventing Unix because that is what we teach. Look across the OSDev Wiki - many articles are written from the perspective of someone who has written a Unix-like operating system. Look at the Forum - many posts here ask for advice on architectural designs and most of the time, Unix is the answer. Heck, look at the tutorials you can use! We've got Xv6, which is a literal descendant of Unix, we've got Minix which is Unix with a microkernel, we've got JamesM's kernel development (Roll your own toy UNIX-clone OS!). My point, dear reader, is that our community teaches new developers to start writing a Unix-like operating system. So, to conclude my main argument, it's not by choice that many programmers write a Unix-like operating system, but simply because it's what's taught. Even if you chose not to, unless you really understand what you're avoiding, you'll end up reusing a lot of Unix material.

Now, you may be thinking "Alright, I see what you mean, but why does this matter?" In all honesty, I don't know. I just felt like posting about it since it seemed like an interesting perspective. I should say that I mean no offense to people writing Unix-like operating systems nor any offense to those who criticize Unix. My take is that Unix has gotten bland and frankly used up in terms of cool things, so my project will get boring quick if I use it. I'd be interested to hear why you did (or didn't) write a Unix-like system, and if you didn't, what ideas did you go for?

[mid]

However, I have a different argument for this: Hobby operating systems, especially beginner operating systems, skew towards poorly reinventing Unix because that is what we teach.

I don't think this is a problem. These are still beginners - they do not know how, nor are confident enough to go outside the Unix methodology, which is pretty intuitive (!= perfect) I think. I guess these pages and guides could certainly have a disclaimer noting that this is one method, but that sort of becomes obvious at some point in a kernel programmer's studies. So, overall I'm neutral.

[bzt]

Personally, I believe that Unix isn't great, but isn't terrible - some design choices are very weird

The same stands for WinNT too (why are all the file APIs doubled with ASCII and UCS2 strings? And what if I want to use more UNICODE code points than what's on the BMP? It is extremely weird, why can't I just use UTF-8 filenames with the ASCII API? (Would be backward compatible BTW)

A specific quote, however, from 1987 on the Usenet, draws my attention: "Those who do not understand Unix are condemned to reinvent it, poorly."

That quote refers to the fact that the UNIX operating system despite being extremely small includes everything that you might need (multi-platform, file abstraction, directory hierarchy, multi-tasking, time-sharing, multi-user etc.), all the OS concepts ever invented. If you think about it, there's nothing new invented since then (just because back then they had one CPU core only and now we have more doesn't change the concept of cooperative multitasking for example, it's a minor implementation detail. Same way, the fact they only had line printers instead of a screen and keyboard, does not change the concept of I/O peripherals, that's just a matter of drivers). Actually, even with the GUI and WIMP paradigm there's nothing new since Eckelbart's famous demo in the 60's.

The hard truth is, IT companies are just selling exactly the same crap over and over again in a different sugar-coat.

Most hobby operating systems lean towards the Unix-like variety. Why is this? I think it can be argued that it's easy to understand, the simplicity makes its implementation easy, or perhaps POSIX is just appealing because of how well defined it is.

These aren't so. UNIX isn't easy to understand (just think about the infamous "You're not supposed to understand this" comment in the source). And POSIX is everything but well-defined. Seriously, strlen() is in string.h, mblen() in stdlib.h and sprintf() is in stdio.h while all of these operate on strings? Or mkdir() in stat.h (what the hell does it do there?) and rmdir() in unistd.h? Why not in stdio.h?

Nope, the only reason why hobby OS choose POSIX because of portability and the sheer number of Open Source projects they could use on their OS.

My point, dear reader, is that our community teaches new developers to start writing a Unix-like operating system.

Yeah, because that's the only option. You don't have the WinNT internals documented, and even if you had, take a look at ReactOS and Wine to see how difficult it is to support that. Not to mention that it's not a wide-spread standard, rather it is the intellectual property of a single company, so you could be sued by M$ any time. There's no such law issue with POSIX standard.

I'd be interested to hear why you did (or didn't) write a Unix-like system, and if you didn't, what ideas did you go for?

You don't really have another option than Unix. Read Andy Tannenbaum's "Modern Operating Systems" book, it makes an attempt to compare Unix and WinNT interfaces to provide an alternative to Unix. My personal opinion is, it fails miserably simply because WinNT internals aren't public or aren't free to use or aren't documented, or aren't standard in the first place and subject to change any time. Go on, buy and read that book and see for yourself!

To design an interface from ground up (neither Unix nor WinNT), that's a task for the real experts only, all beginners will fail in that. And even if you manage that somehow, you'll have to rewrite all applications yourself, because you won't be able to compile Open Source projects for your OS. But not impossible, see MenuetOS (and you should learn why it was forked into KolibriOS). You should also learn about SkyOS, which had it's own interface too, and why it was discontinued (The wiki is wrong about SkyOS being unix-like, read this interview with Szeleney).

Cheers,
bzt

[rdos]

You don't really have another option than Unix. Read Andy Tannenbaum's "Modern Operating Systems" book, it makes an attempt to compare Unix and WinNT interfaces to provide an alternative to Unix. My personal opinion is, it fails miserably simply because WinNT internals aren't public or aren't free to use or aren't documented, or aren't standard in the first place and subject to change any time. Go on, buy and read that book and see for yourself!

I read that book mostly for fun many years ago, and I certainly was not impressed by Unix and didn't want to do something that already had been done.

That's a bit how it is. There is no use in writing a Unix-clone or a Windows-clone since it will not be better than the original anyway. I was motivated to start my OS based on how horribly bad DOS was when I came in contact with it in the late 80s. I certainly didn't want to write a clone of that horrible design, but I had the idea to be compatible with it to some degree. Since then, I've had Win32 emulations that were good enough to fool the Borland Debugger, and I have a bit of the Posix interface too (most notably, fork). But I would not even dream about basing my design on Unix, DOS or WinNT. All of these are poorly designed OSes.

To design an interface from ground up (neither Unix nor WinNT), that's a task for the real experts only, all beginners will fail in that.

I've certainly done that. I've had a couple of major rewrites, but that should be normal for any large software project.

And even if you manage that somehow, you'll have to rewrite all applications yourself, because you won't be able to compile Open Source projects for your OS.

I think that is a pretty poor argument. The problem is that if you are not exceptionally compatible with Linux or Windows, then you will not be able to compile a majority of these even if you have a full libc with Posix. Most of these projects are full of ifdefs and typically require autoconf to even generate makefiles. If you manage to fix the ifdef part so it builds for your OS, chances are you will not be able to commit your changes to the project and so your modifications will not follow the projects and so you need to redo them regularly.

For the projects I've ported, like JPG, PNG and SSL, I've added them to my own SVN repository and then only update them quite seldom.

[AndrewAPrice]

The way I see it, we are all part of a baking community and most people are baking chocolate cake, and many of the baking tutorials are pushing you that's learning how to bake chocolate cake. There's no shame in baking a chocolate cake, but some people would rather bake a cheesecake, and otherwise would rather go totally different and bake an apple pie.

I'm not building a Unix clone because I have too many ideas of what I want to do. For example:

• All IPC in my microkernel is done by implementing and discovering instances of named services that are generated from an IDL (example). The messages are typed, dynamically sized, and are sent by transfering the memory pages between processes so there's 0 copying.
• Per-program capabilities based rather than per-user permissions (e.g. "Is Chrome allowed to play audio?" "Is Blender allowed to read files under /Documents/Renderings/?" "[Always] [Just this once] [Never]") My personal documents are more valuable to me than system files that I can reinstall.
• A window manager where windows can be docked and tabbed (example).
• A different file system hierarchy. Programs live in an "Applications" directory and are self contained. No /bin /sbin/ /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin.
• A custom C++ build system. I shouldn't need a build script when all I want to do is throw a bunch of C++ sources in a folder, and a list of libraries I depend on. It worked enough to build Musl and Libc++. (My metadata file for building Musl.)

I have ported a C and C++ standard library. One day I plan to port SDL (it seems easy enough.) I know I'm not going to be compiling programs built for WinAPI or xlib. I know I'm not having driver compatibility with any other OS. I'm ok with that. That's the fun of operating system development.

I also know that not all my ideas are unique, and there's likely some 1960s OS that does everything above and more. I don't care, I'm building these things for the fun of it.

[andrew_w]

I'm writing a Unix-like system because I'm wanting something that's practical and can eventually become my main OS, and I'm very familiar with Unix-like systems, rather than because I think Unix is perfect. IMO new non-Unix-like systems are usually in the category of interesting but impractical. If practicality isn't a priority for you, then I say go ahead and write a non-Unix-like OS, but don't expect it to become a runaway success in terms of widespread usage.

And I'd say there are still some interesting ways to write a Unix-like OS. UX/RT, the OS that I'm writing, will have a rather uncommon architecture with a few features that I have never seen anywhere else. Specifically, it will be a QNX-like system, of which I am aware of only a few others that have ever been finished (QNX itself, VSTa, and Helios, although Helios may have been designed completely independently of QNX). The biggest completely new (AFAIK) feature that UX/RT will have as far as the fundamental design goes is that it will take "everything is a file" as far as it can be taken. It will take this even further than Plan 9 does (anonymous memory will be replaced with mapped files in a tmpfs, processes and threads will be created through procfs, and even most of the filesystem API itself other than read/write/seek will be implemented with RPCs sent over a permanently open file descriptor to the VFS). I'm also going to fix a lot of other shortcomings of legacy Unix as best I can without breaking compatibility too much (the goal is to run most Linux programs except for those that either manage sessions/logins or depend on a specific kernel module that can't be run in LKL).

[iansjack]

On Side A, we have the Unix haters... On Side B, we have the Unix lovers

Really. Aren't we more grown up than to talk in these sort of terms? This is like Microsoft/Apple or PS/XBox flame wars. Software either does the job it is designed to do well, or it doesn't. Unix does what it was designed to do supremely well.

On a high level, most operating systems adopt the same principles as Unix - files, folders, network sockets, system calls, devices as files, etc. Unix - in it's various guises - is documented in a host of books. The reason that people start off by implementing a Unix-like OS, the reason that so many tutorials concentrate on such design, is that it is universally popular, simple (in it's simplest guise), and well documented.

We don't use Unix as a model because that's what the tutorials do; the tutorials use Unix as a model because it is so widespread and popular.

But we would welcome a tutorial from you on alternative design paradigms.

[nexos]

On Side A, we have the Unix haters... On Side B, we have the Unix lovers

Really. Aren't we more grown up than to talk in these sort of terms? This is like Microsoft/Apple or PS/XBox flame wars. Software either does the job it is designed to do well, or it doesn't. Unix does what it was designed to do supremely well.

On a high level, most operating systems adopt the same principles as Unix - files, folders, network sockets, system calls, devices as files, etc. Unix - in it's various guises - is documented in a host of books. The reason that people start off by implementing a Unix-like OS, the reason that so many tutorials concentrate on such design, is that it is universally popular, simple (in it's simplest guise), and well documented.

We don't use Unix as a model because that's what the tutorials do; the tutorials use Unix as a model because it is so widespread and popular.

But we would welcome a tutorial from you on alternative design paradigms.

I completely agree. Most OS concepts have come from Unix. Unix is sort of the "parent" of all modern operating systems, NT included. IMO, NT is much worse then Unix. The object manager can be confusing (Are objects files? Does the file system count as an object? Why are they separated?). The Unix method is very powerful. Unix has been made bloated, but, as it was said "Those who do not understand Unix are condemned to reinvent it, poorly." I hate to say it, but modern Unixes are written poorly, and are not what Unix was designed like.

My OS is going to be Unix in a way. I plan on moving everything into the filesystem, that way, security is more centralized. The API will be Unix compatible, but Unix is going to be implemented in a server like Managarm does it. Plus, it will all be in a microkernel based system.

[zaval]

nexos, c'mon, UNIX didn't create our Universe. It's just a castrated implementation of MULTICS (just say "eunuchs", see, it's their own joke). The only good thing derived from UNIX is C. I agree with the author, that the reason why UNIX cloning is so dominant is due to universities teach exactly UNIX, MINIX etc. /shrugs/

[vvaltchev]

I believe that UNIX, while not perfect, is good. It influenced the design of all the successful operating systems, including NT. I don't think there's anything wrong in teaching it: we don't have anything dramatically different from it that's really successful except NT but: 1) it's proprietary 2) it's very big and complex 3) still have some things in common with UNIX and it's actually moving closer to them, not further away (see WSL). Of the 3 major operating systems we have today (Linux, NT and MacOS), two of them are part of the *NIX family. If I had to mention a 4th operating system, I'd say FreeBSD, which is still UNIX, so 3 out of 4, and that's without counting operating systems for mobile. Given how much Microsoft is investing in Linux, I'd guess it's even possible that one day the NT kernel will be replaced by a sort of Microsoft-patched Linux kernel.

Anyway, my point was not to praise UNIX (which have its own weak points), but to point out that inventing something completely different is HARD, extremely hard. That's also in part because of the hardware too, which is designed for operating systems to work in a certain way (e.g. the flat-memory model etc.). If anybody will be so smart to invent a new paradigm, I'd bet that person will be an expert with many years of experience in the field, not someone just approaching it. More likely, a team of brilliant minds. So, it's pretty normal that here a lot of people will write UNIX-like kernels. People tend to first learn the state of the art before thinking about how to change it, and that's IMHO a sane approach, in general for everything.

[zaval]

It influenced the design of all the successful operating systems, including NT.

Given David Cutler's dislike to UNIX, that's hardly a more, than far fetched, wishful thinking. But yeah, since noone really knows, what the "UNIX philosophy" apart from banging keyboard in vim/emacs, telling "everything is file" mantra and calling administrator account "root" is, then with a bit of will, one can proclaim even the earlier appeared OSs, "influenced" by UNIX.

Given how much Microsoft is investing in Linux, I'd guess it's even possible that one day the NT kernel will be replaced by a sort of Microsoft-patched Linux kernel.

They offload spendings using it, as every corporate linux "lovers" do, but why would they replace a far superior, fully fledged, ready to keep its dominance, with such the heritage and ecosystem OS, remains the question. Why will they want to replace their crown jewel with a plastic substitute? The code base, they have been nurturing for decades, the code base they fully own and don't need to pull off from somewhere outside and "patch" that mess to fit their visions. Windows is not IE/Edge. Even that was a real bad decision, hurting more a very deplorable browser situation. Without real competition, browsers will get so "improved" soon, that will be eating up gigabytes of memory for an empty page. Not hundreds of megabytes as is now. But Windows is not the case. Like at even 0.001% of chance. Maybe it's just wet fantasies of linux fans, so hardly trying to show, they don't care about anything Windows, yet generating assumptions like this? Anyway, if that happens, then this will be the dumbest decision ever made in this industry, and given MS is always #1 in everything, who knows. Heil ReactOS then!

PS. While linuxoids, waiting for the "year of linux on desktop" (for 20+ years so far ), are tasting this kind of day dreaming, "replacing" NT with linux in their heads, I bet, The Corporation Of Goodness will replace the linux kernel in android for real and linux won't be a "runs on 100500 billions of trillions of phones!!111" thing anymore. So the only "It's seeervaz!" chant remains.

[nullplan]

nexos, c'mon, UNIX didn't create our Universe. It's just a castrated implementation of MULTICS (just say "eunuchs", see, it's their own joke). The only good thing derived from UNIX is C

Yes, the name may have been a self-deprecating joke, but UNIX is far more functional than MULTICS ever was. One good thing to come out of UNIX is the pipe. MULTICS had pipes, but it didn't have the convenient syntax of "command1 | command2". Instead you had to issue multiple commands to set up a pipeline and it was usually more hassle than worth.

And given that pipelines are now part of all operating systems I can think of (including MS-DOS, although its implementation was... special), I'd say they were a very good thing to come out if UNIX.

Of the 3 major operating systems we have today (Linux, NT and MacOS), two of them are part of the *NIX family.

Isn't MacOS running atop a kernel called XNU, which is very Unix-like?

Given how much Microsoft is investing in Linux, I'd guess it's even possible that one day the NT kernel will be replaced by a sort of Microsoft-patched Linux kernel.

I've heard that before, and... no. No, I don't think that will ever happen. First of all, who cares about the kernel underlying the OS? Hackers. Not normal people. For such a kernel replacement there are only two ways: Make it so that people notice or make it so that people don't notice. If people notice, it will likely be because something isn't working, which is not exactly profit-inducing. If people don't notice, that means MS spent tons of effort porting everything and creating compatibility layers, so that existing applications run. But why would they spend such effort on something that is designed not to be noticed? MS is a for-profit company, they only spend money if it can make them money in the long run. And this kernel swap that people envision could only cost them money.

Plus, NT has features Linux doesn't. It is my understanding that the main NT kernel is a hypervisor, and the various subsystems run as guest operating systems. That is why it was possible to just make a Linux kernel run as yet another subsystem. NT, as it is now, is working fine for what MS is doing with it. There is little impetus to change it.

[rdos]

nexos, c'mon, UNIX didn't create our Universe. It's just a castrated implementation of MULTICS (just say "eunuchs", see, it's their own joke). The only good thing derived from UNIX is C. I agree with the author, that the reason why UNIX cloning is so dominant is due to universities teach exactly UNIX, MINIX etc. /shrugs/

Kind of, and I have to disagree with C being something good too. I don't use C anywhere (except for in some complicated device-drivers), as my view is that either you use assembler or C++. C is a kind of middle-of-the-road alternative that is poor at both low-level stuff and object-oriented programming, and so has no place in my design.

[rdos]

Given David Cutler's dislike to UNIX, that's hardly a more, than far fetched, wishful thinking. But yeah, since noone really knows, what the "UNIX philosophy" apart from banging keyboard in vim/emacs, telling "everything is file" mantra and calling administrator account "root" is, then with a bit of will, one can proclaim even the earlier appeared OSs, "influenced" by UNIX.

It's a good thing I have none. I think the user account model is outdated, and the ancient terminal emulators are of no interest anymore.

[rdos]

I completely agree. Most OS concepts have come from Unix. Unix is sort of the "parent" of all modern operating systems, NT included. IMO, NT is much worse then Unix. The object manager can be confusing (Are objects files? Does the file system count as an object? Why are they separated?). The Unix method is very powerful. Unix has been made bloated, but, as it was said "Those who do not understand Unix are condemned to reinvent it, poorly." I hate to say it, but modern Unixes are written poorly, and are not what Unix was designed like.

My OS is going to be Unix in a way. I plan on moving everything into the filesystem, that way, security is more centralized. The API will be Unix compatible, but Unix is going to be implemented in a server like Managarm does it. Plus, it will all be in a microkernel based system.

The everything is a file concept that I suppose Unix invented is not a great invention. Mostly because they also added the ioctl interface for everything that is not file-related. Which means that every device invents its own ioctl interface that typically is undocumented and incompatible with every other device. The "device-tree" in the file system might seem like a smart idea, but the names are not standardized and it hinders more efficient implementations than just read/write passing buffers. The fork() API is not a great invention either and gives a lot of headaches for people trying to implement this on multicore systems. It might have seemed smart when it was invented, but today it's not smart. The CreateProcess of Windows is far better.

In short, Unix is a design of the 70s, and as such is pretty poor in the 2020s when the hardware is quite different.