Is OS development in threat of extinction?

[austanss]

This might sound preposterous, but hear me out.

Currently, all the wiki content and various OS tutorials reach a certain level of OS advancement. Past that, it’s basically mandatory that you learn how to RTFM. This isn’t a problem in itself, though.

The current trend of the OS development community makes it seem like the field of OS development is rapidly expanding. However, that is simply quantity over quality.

Exhibit A: as of posting, the last post on the “OS Design and Theory” forum was a month ago. The latest post on the “OS Development” forum (where people usually ask for help) was today. Guess what the topic of the post was? It’s an easy guess. paging

Now obviously, I am not innocent in this regard at all. A couple years ago, I was no better than anyone else who asks for PAE help on this forum.

Why is this a problem? It seems that, day by day, the OS development scene is flooded with more novices than it can accommodate. Most of the senior OS developer types, the “I was there when FSF was born” types, will in the next ten years either retire, die, or cognitively decline.
These senior OS developers have been carrying the entire hobbyist scene on their backs with virtually no compensation or recognition.

The typical OS developer these days follows a typical routine. 1. Do the Bare Bones. 2. Install Limine. 3. Struggle with basic CPU and IO configuration. 4. If they make it past that hurdle, then they write a scheduler, copying UNIX kernel designs. 5. Maybe port a libc. 6. Maybe port gcc. 7. Sit back and do nothing.

This is why Terry Davis, for all his faults, deserves recognition as a savant OS developer. He wrote the HolyC compiler, then wrote TempleOS completely from scratch (no bootloader or libc), and not only did it all alone, but he made an extremely unique (if not limited) design that was actually reasonably sensible.

My main point is that all of the extremely experienced OS developers are fading away, replaced by a new generation of socially incompetent, narcissistic teenagers who do not actually fully understand the concepts of OS development and will likely not be competent enough to keep the OS development community alive and healthy.

I do not intend on arguing this from a position of superiority. I simply argue this from a perspective that I perceive to be full of clarity. I am definitely no senior OS developer, and none of my work measures up to what most moderators on these forums have accomplished. Yet, I worry that, in due enough time, the knowledge and wisdom that is required to become a OS developer with the same skill level as a London cab driver will simply disappear alongside the prolifically technical minds who harbor them.

Thoughts?

[BigBuda]

What you said is not preposterous at all and I've been thinking about the exact same thing since the end of the 90s, when I started realizing that architectures are becoming more complex and documentation on the hardware more closed at each cycle. The level of complexity of a computer nowadays, even a small one like a mobile phone, is staggering. Things have evolved, not always for the best. Besides, a lot of the complexity nowadays stems for a (sometimes pathological) need for backwards compatibility. ARM didn't make things easier, as the architecture has adopted (and adapted) many of decisions (good and bad) of other architectures. Which, in part, makes sense, but it also has drawbacks. The x86-64, which appeared around 2002, if I remember correctly, also didn't make things easier, just added more complexity on top of an already complex device and it's all been downhill from there.

[austanss]

What you said is not preposterous at all and I've been thinking about the exact same thing since the end of the 90s, when I started realizing that architectures are becoming more complex and documentation on the hardware more closed at each cycle.

A possible lack of documentation, if reality, would put all OS development expertise in the hands of a select corporate few, ultimately autocratizing all computing for good. That really scares me.

[BigBuda]

A possible lack of documentation, if reality, would put all OS development expertise in the hands of a select corporate few, ultimately autocratizing all computing for good. That really scares me.

Which is exactly what we've been seeing for a long time. The time to be scared with that future is over. Now is the time to panic because it became reality. Fortunately, there still some ingenious people out there able to somehow reverse engineer a lot. I can't be sure, I don't have statistics and not even anecdotal evidence, but I have a deep chilling feeling that most of what is considered public knowledge nowadays was acquired this way.

[eekee]

I stopped panicking years ago. Why? Because I kept finding surprising examples of openness. I also have doubts that internally-divided, competitive organizations can maintain authoritarianism for long. In business particularly, authoritarianism is quick to evaporate when it gets in the way of profit. Consider Apple's changes over the years; switching from entirely closed firmware to OpenFirmware, adopting PCI and PCIe, and even switching to a nearly-standard PC platform. Note too the many contributions to open source by the worst of authoritarian companies -- Apple and Microsoft. I've also seen reverse engineering increase the more closed hardware appears.

I'm too short on sleep to write anything in-depth, but I think there's 2 issues facing us.

1) Rising complexity. This seems to be an inescapable part of increasing power, but is it? And how much power do you need for your OS, anyway? I remember Linux doing everything I wanted with 64MB RAM and a 120MHz Pentium pro. I can certainly get quite a lot of useful work out of that hardware or even much less, and I can both innovate and enjoy my hobby. This is practically embedded hardware level these days, and embedded hardware has to be documented for the developers. Of course, I've wanted more since that old Pentium; specifically 3D and video, and I've wished for 4k resolution. The situation with these features is less encouraging, but some video codecs are better than others and hardware support is possible.

2) The glacial but inevitable demise of the PC. This one might not even be real in the sense that the situation the PC created was never as good as it seemed. The very compatibility we blame for complexity has enabled a bloom of OS development as much more has been written about the PC than any other platform. Nearly all new OSs get written for the PC platform, but then we find they run only in a handful of emulators and maybe 1 or 2 very specific laptops. The PC's openness has enabled a profusion of perpherals which need different drivers, but even with as central a component as the BIOS, "Booting on one computer is easy; booting on many computers is hard." UEFI doesn't seem to be much better in that regard.

Also remember that the platform which is, for real, emerging as the "one true OS" is POSIX which is not owned by any one company.

On a related note, how many of us have full control of our Linux systems and still have time for OS development? I've resigned myself to maintaining "ugly boxes" with a black-box OS -- Linux counts as black-box in this context just as much as Windows and Android -- while developing and running more accessible and interesting systems on other hardware and emulators. The ugly boxes can handle all the 3D, video, and most of the high-quality font rendering I need.

I want to target hardware which users can readily and affordably obtain in consistent form. I'm not yet sure which hardware, from boards marketed to hobbyists to a custom PCB supporting one of those ARM+FPGA SoCs. 9front devs made their "aijuboard" around a Xilinx ARM+FPGA SoC which supported SATA, framebuffer graphics, real ethernet and of course USB, and that was 9 years ago now.

tl;dr: We can't have everything we wish for, but the situation's not all bad. There is plenty of opportunity for OS dev.

[eekee]

Published on Youtube while I was typing all that:
Corporate Open Source is Dead

Nobody likes being rugpulled. But lately, it's going around like a virus.

Why are so many former open source darlings selling out or relicensing? And is there anything you can do to fight back against these anti-open-source practices?

I think my point stands, but I thought I ought to link this. Haven't watched it; too tired, but Jeff Geerling seems like a sensible guy.

[austanss]

Published on Youtube while I was typing all that:
Corporate Open Source is Dead

I have never heard of this rugpull phenomenon before. Pro tip guys, never sign a CLA.