Python OS

[klange]

Perhaps I am being too cryptic here, so to be a bit more clear: I built my own implementation of (a dialect of) Python, and a bit over a year ago I put some effort into porting it to run freestanding and under EFI, as I wanted to use it to build a highly configurable bootloader. I abandoned the freestanding port, but I maintain the EFI version.

One of the pros you listed for Python was garbage collection, but you don't get garbage collection for free, you have to implement the garbage collector yourself (as part of your Python runtime, even if the Python is compiled to native code) on top of all the other technical challenges of building an OS.

CPython traditionally didn't have a garbage collector, in a pedantic sense. It's primarily reference counted, so all you really need is a good ol' heap. (It has a garbage collector these days - gotta deal with those pesky reference cycles somehow! And some other implementations, notably PyPy, are garbage collected in the traditional "mark and sweep" sort of way.)

The Python interpreter (CPython) doesn't support multi-threading.

This is a controversial statement, and I think in the context of "building an OS" it really depends on how you're using Python - and what your target is. For a single-processor environment, even if you're going for a Cosmos-style "managed code" system where user processes are running on a shared interpreter (which is a silly idea - Python is not a remotely safe VM), the GIL is meaningless - you can't run multiple threads simultaneously on a single core regardless. If we're talking SMP-capable with a "pure Python" kernel and device drivers, it's going to be a big oof, but I think there's a solid middleground where even CPython could operate as a glue layer managing processes and drivers running natively. The GIL can be released when running native code - a very critical thing anyone who persists in trying to do multi-threaded things in CPython should be well aware of - it's why frameworks like NumPy can provided the performance they do. But why even share an interpreter in the first place? You could run a separate interpreter for each core, share specialized resources between them with finer-grained locks, and not have to worry about any of this.

It is a lot of work, arguably more work than writing a kernel.

Eh, my kernels were way more effort. Python is not a very large language at the interpreter level.

[kzinti]

This is a controversial statement, and I think in the context of "building an OS" it really depends on how you're using Python

It depends on what is meant by writing the kernel in Python. I took this literally to mean that almost everything in the core is written in Python, including the scheduler, memory management and interrupt handlers. The GIL would then be a serious problem. If you are going to write the core in C/C++/assembly and add Python drivers on top, I would argue that you are not writing a Python OS anymore.

Eh, my kernels were way more effort. Python is not a very large language at the interpreter level.

It depends on the scope of your kernel and the scope of your Python interpreter. I implemented a full Python interpreter myself and was trying to get a JIT compiler going (one similar to LuaJIT) and ended up abandoning it because I didn't have enough time to dedicate to that project. I suppose if all you do is interpret byte code like CPython does, it is not that much work. Also I was assuming that the kernel here was a microkernel. If you include all the drivers and services you need on top of it, then I agree that writing a Python interpreter is much less work.

But here is the thing, I have the feeling that someone asking if you can write an OS in Python is asking so because they don't feel comfortable with C/C++/assembly and want an "easy way" to write their OS using a language they are familiar with. If this is indeed the case, than using Python will likely end up being more work as not only will one need to understand the internals of a Python interpreter, but one will still need to understand how to write a kernel. There is no shortcut here.

[eekee]

This all reminds me of the time I built Python to run on top of a bare Linux kernel only. One of several projects I should have persisted with in the mid-00s.

but that's not "Python for OS development"

No, look at all of the C# OSes out there. They have a tool which converts IL to native code. Same goes for Python.

So, if I'm not mistaken, one route would be IronPython which compiles to .NET, then this IL to native code transpiler. Or, I guess PyPy, assuming its JIT-compiler can be persuaded to produce code files.

The Python interpreter (CPython) doesn't support multi-threading. That would be a huge impediment to writing on OS in Python. You could try to adapt PyPy, but that would be an enormous undertaking.

Er... I wrote a Python program with 2 threads back in the early 00s. More specifically, it was 2 processes, but Linux hardly distinguished between threads and processes in those days; every thread had a PID. (I still prefer the idea of lightweight processes to making threads a different kind of entity.)

@klange: Kuroko under EFI sounds like fun! If I hadn't just started a native Forth project, I'd have a play with it.

[eekee]

I forgot, they use Python on supercomputers; CPython. I'm not sure of the details, but I think it's partly that they just tolerate the overhead, and partly that Python is suitable for functional programming since functions are first-class objects. Having said that, functional programming is a poor choice for some tasks.

[Demindiro]

Er... I wrote a Python program with 2 threads back in the early 00s. More specifically, it was 2 processes, but Linux hardly distinguished between threads and processes in those days; every thread had a PID. (I still prefer the idea of lightweight processes to making threads a different kind of entity.)

The Linux kernel still considers processes and threads to be the same thing (a "task") and each thread gets their own PID.
IMO it is a very poor idea to combine both since processes and threads store very different data. In fact, combining them probably makes processes/threads heavier since you now need a way to share address spaces / object sets ... instead of letting a single process object own them.
I also get an aneurysm every time I read the documentation for clone(2).

BTW, if you want to run multiple "threads" concurrently (i.e. without being hindered by the GIL) you should look into the multiprocessing library.

I forgot, they use Python on supercomputers; CPython. I'm not sure of the details, but I think it's partly that they just tolerate the overhead, and partly that Python is suitable for functional programming since functions are first-class objects. Having said that, functional programming is a poor choice for some tasks.

I imagine the Python code only calls optimized routines (and hence only takes up a fraction of the time) since otherwise it's a huge waste of energy, especially if you use CPython instead of e.g. PyPy.

[kzinti]

Er... I wrote a Python program with 2 threads back in the early 00s. More specifically, it was 2 processes, but Linux hardly distinguished between threads and processes in those days; every thread had a PID. (I still prefer the idea of lightweight processes to making threads a different kind of entity.)

Assuming Python threads and CPython, it doesn't matter how many threads you create: only one of them can execute at a time. Libraries such as NumPy can get around this limitation by using native (worker) threads. Asynchronous operations can also be done natively by library/module code to work around this limitation (think disk I/O and networking for example).

If you are running two Python instances, then you are running two programs.

[klange]

Assuming Python threads and CPython, it doesn't matter how many threads you create: only one of them can execute at a time.

This is an inaccurate description of how the GIL in CPython works. "Python threads" are real, native OS threads. The GIL prevents large parts of the interpreter itself from running in parallel, including (rather critically) the main eval loop, but C extensions are allowed to release the GIL as they perform other work.

Libraries such as NumPy can get around this limitation by using native (worker) threads.

NumPy only uses worker threads indirectly through a linear algebra backend. For everything else, NumPy releases the GIL around thread-safe native operations, which allows multiple "Python threads" to make calls to these functions and have them execute in parallel.

[kzinti]

This is an inaccurate description of how the GIL in CPython works. "Python threads" are real, native OS threads. The GIL prevents large parts of the interpreter itself from running in parallel, including (rather critically) the main eval loop, but C extensions are allowed to release the GIL as they perform other work.

This is correct. In practice only one Python thread is running at a time unless you are using C extensions that can release the GIL for significant amounts of time. This means you need to find work that can be batched for processing or work that can be completed asynchronously and implement them as C extensions. If you are planning to write your OS as C extensions, you are not writing your OS in Python but in C and using Python to glue things together. This could be a reasonable approach.

My argument is that if you want an efficient OS written in Python, you will not be saving any time over writing it in C. You still need to understand all the low level technicalities and still need to implement your core components in C. On top of that, you also have to integrate or implement a Python interpreter.

[immibis]

If you dislike the idea of a huge C or assembly runtime, another option is to invent a "Systems Python" language that is basically a variant of Python with the minimal amount of work to make it statically typed/manual memory management while still having the look and feel of Python. Bonus points if your code is actually valid Python too.

Write your core microkerne/runtime n "Systems Python" with all applications and services in regular Python.

AFAIK the pypy interpreter is built this way. It's a Python implementation written in Python. How's that work? It's actually written in RPython - Restricted Python - a subset of Python which is compiled by their custom toolchain. In fact, the compiler starts out by running the interpreter as regular Python, so it's able to dynamically create classes and functions and stuff, and then a certain library call marks the point where the compiler stops running the code, and compiles whatever objects are in memory at that point. Code after that point - because it is compiled and runs at runtime - can't do things the compiler doesn't support.

[eekee]

Assuming Python threads and CPython...

I don't think Python had threads when I wrote it. POSIX didn't standardize threads if I remember rightly, and I'm not sure how mature (or even extant) the pthreads library was.

[devc1]

BRUUUUV, WTF ??? Do you plan to use the slowest, most bloated and dumb language on earth to make an OS. Go learn some C, I started without understanding what are bitwises. Python is what makes you need to buy a 10000$ PC to run applications smoothly : )

In C you can get external Assembly Acceleration, Optimize with SIMD parallel processing, create your own memory manager, your own process manager. By time, you will know the assembly output of each line of code you wrote in C, this is impossible with other (dumb) languages.

Make me wrong if you can ! Python is the dumbest creation on this earth.

Just face the truth, you can do it but in terms of capabilities and performance MS DOS will still beat you.

Devc1,

[Schol-R-LEA]

BRUUUUV, WTF ??? Do you plan to use the slowest, most bloated and dumb language on earth to make an OS.

If you think that Python is slow and bloated, I hope you never end up working in Haskell, Smalltalk, or Prolog. Or even Eiffel, which despite being natively compiled for Windows is notorious for producing a 2 Megabyte "Hello, World!" which takes several seconds to run (though that was more about the specific compiler than about the language - other implementations do much better).

And yes, I know that this must seem rich coming from a Lisp Weenie such as myself, but despite the reputation generated by the innumerable interpreters out there, there do exist some high-performance Lisp compilers. Not that you would know it from, say, Dr. Racket...

But raw performance is not the be-all and end-all of programming. Having an expressive language with a rich library and features such as automatic memory management and support for OOP makes everyday programming much easier. While in theory, any Turing-complete language has the same computational power, in practice languages differ in expressivity - otherwise, specialized languages such as SQL wouldn't exists. Not every programming project needs to be done at the lowest level. Performance, like everything else, is about compromise. For its intended roles, Python does pretty well.

For that matter, C is far from the only "high-performance language". While C++, D, and Rust generally do come with some performance cost compared to C, what they bring to the table in terms of language features more than makes up for it to those using them. And there's even Forth, which is in a category of its own and doesn't really resemble anything else - despite being an interpreted language (sort of), it often outperforms much C code out there.

In any case, as has been said elsewhere, it is just as possible to write poorly performing code in C and assembly language as in any other language. Programming skill is a bigger factor in most cases than the implementation language is.

[iansjack]

I’m afraid that it’s a common misconception amongst people who have no experience of real-world software projects that performance is all that matters. In reality it is relatively unimportant compared with ease of implementation and maintainability. It’s much more cost-effective to throw more hardware at performance problems rather than waste time chasing after kilobytes of RAM or tens of clock cycles.

Sure it’s good to write efficient programs, but not at the expense of maintenance in particular.

[Schol-R-LEA]

I’m afraid that it’s a common misconception amongst people who have no experience of real-world software projects that performance is all that matters. In reality it is relatively unimportant compared with ease of implementation and maintainability. It’s much more cost-effective to throw more hardware at performance problems rather than waste time chasing after kilobytes of RAM or tens of clock cycles.

Sure it’s good to write efficient programs, but not at the expense of maintenance in particular.

Yes. It is a matter of balancing priorities, really.

Unfortunately, many managers seem to focus on one aspect of a project over all of the others, and while the most common place to sacrifice is in the testing and QA, I have seen some over-focus on performance at the expense of time overruns and poor maintainability.

Worse, many managers get caught up in fads and buzzwords, rather than actually looking at what is the best fit for the job. But that's getting away from the topic.

[eekee]

I’m afraid that it’s a common misconception amongst people who have no experience of real-world software projects that performance is all that matters. In reality it is relatively unimportant compared with ease of implementation and maintainability. It’s much more cost-effective to throw more hardware at performance problems rather than waste time chasing after kilobytes of RAM or tens of clock cycles.

Sure it’s good to write efficient programs, but not at the expense of maintenance in particular.

It's much more time- and effort-efficient for hobbyists, too.

I do wonder how efficient Python can be, because it's been in widespread use in ATMs for maybe a decade now.