Opinions on scheduler (C++ coroutines and NodeJS)

[bellezzasolo]

I want full control of IRQs, and using C, C++, exception handling and co-routines in IRQs seems like a nightmare to me.

The way I see it all the IRQ handler generally should do is schedule the lower half and some housekeeping. Currently my mechanism for doing that is signalling a sempahore and a thread.

Coroutines mean you can instead have a kernel thread that deals with deferred IRQ handlers and higher priority than other threads. So the IRQ context switches to that thread if it's not already active.

You then get the nice ability that the programming model for the lower half can easily invoke asyncrhonous functions, and other work gets scheduled.

[rdos]

I want full control of IRQs, and using C, C++, exception handling and co-routines in IRQs seems like a nightmare to me.

The way I see it all the IRQ handler generally should do is schedule the lower half and some housekeeping. Currently my mechanism for doing that is signalling a sempahore and a thread.

Coroutines mean you can instead have a kernel thread that deals with deferred IRQ handlers and higher priority than other threads. So the IRQ context switches to that thread if it's not already active.

You then get the nice ability that the programming model for the lower half can easily invoke asyncrhonous functions, and other work gets scheduled.

A majority of my IRQs will "signal" a kernel server thread to do the actual job. The kernel thread don't need to worry about blocking like IRQs need to. The signal function is designed so the thread is always activated, regardless if it is blocked, about to be blocked or running. The thread control block has a flag that keps the signalled state, so no semaphore is needed. It's implemented as a basic scheduler/task primitive that works on multicore and can be called from IRQs. Actually, it's the only scheduler function that an IRQ is allowed to call.

[AndrewAPrice]

I want full control of IRQs, and using C, C++, exception handling and co-routines in IRQs seems like a nightmare to me.

I don't use C++ in the kernel IRQ handler if that's what you mean. My IRQ handler runs in the kernel and just sends a message to the process that said it's interested in that IRQ.

[rdos]

I want full control of IRQs, and using C, C++, exception handling and co-routines in IRQs seems like a nightmare to me.

I don't use C++ in the kernel IRQ handler if that's what you mean. My IRQ handler runs in the kernel and just sends a message to the process that said it's interested in that IRQ.

That's a bit like the kernel server thread solution, but with longer latency.

Most modern devices will work with these solutions, while some legacy stuff won't. For instance, I don't think you can have server threads with serial ports. These will typically require reading out & buffering characters in the IRQ. Same thing might apply for PS/2 keyboard & mouse as well.

[Octocontrabass]

Most modern devices will work with these solutions, while some legacy stuff won't. For instance, I don't think you can have server threads with serial ports. These will typically require reading out & buffering characters in the IRQ. Same thing might apply for PS/2 keyboard & mouse as well.

I don't see why it wouldn't work. Actually, it should be easier than other devices since they're always edge-triggered and can't raise a new IRQ until the driver acknowledges them.

[ArsenArsen]

Well, you typically want to enable & disable interrupts in IRQs, you might need spin-locks, and AFAIK, C doesn't support any of that without various hacks.

If you really want to avoid using non-standard inline assembly extensions, you can call a separate function written entirely in assembly. Linking C and assembly together is not a hack.

Spin locks can be implemented using atomic_flag from <stdatomic.h>, which has been part of C since C11.

I prefer to code this in assembly, not in C. There is no advantage of having IRQ handlers or spinlocks in C. They will be cluttered by stuff that make them more unreadable than pure assembly code.

you don't need any tricks or 'fancy constructs' here. the aforementioned atomics work trivially and cleanly.

i'd go as far as to say that asm ("cli") is neither a hack nor unclean. it conveys a clear purpose. of course, you should hide such an architecture specific detail behind a layer of indirection. ditto for many other cases of inline assembly, especially extended inline assembly.

My opinion is that if you cannot write C code without using tricks or fancy constructs, write it in assembly instead. Atomic variables or variables where the optimizer is not allowed to remove references are good examples of code that is better done in assembly. Simply because the assembler won't try to remove constructs it doesn't find useful.

compilers don't just decide some code is not useful. if the compiler removed a reference, it wasn't needed, and you failed to convey your intention in your code. it'd have confused a human reader too.

The discussion was more about C++, where you never know the side-effects of even trivial code. C without exception handling is more appropriate, as then you know the side effects of the code without needing to look at the generated code.

that is not how this works. c++ code that you write does what you write it to do.

again, exceptions are neither required nor impossible to disable (nor specific to c++. -fexceptions works for C)

In fact, in the C based drivers I have, I decided that the syscall interface is better done in assembly, and then I would define C based handler procedures with a specific register call interface. That works well for complicated drivers like hid, audio codecs and font utilities.

it is not particularly odd to write stubs in assembly. your original description makes it sound like the entirety of your irq code is in assembly, which'd be an utter waste of time.

[rdos]

Most modern devices will work with these solutions, while some legacy stuff won't. For instance, I don't think you can have server threads with serial ports. These will typically require reading out & buffering characters in the IRQ. Same thing might apply for PS/2 keyboard & mouse as well.

I don't see why it wouldn't work. Actually, it should be easier than other devices since they're always edge-triggered and can't raise a new IRQ until the driver acknowledges them.

When a new one is raised (because another character arrives), the old one is lost if the port has no FIFO. So, the issue is that with high enough baudrate, and no (or small) FIFO, characters will be lost if not handled fast enough. The fastest handling of course is buffering characters in an IRQ written in assembly.

[rdos]

Spin locks can be implemented using atomic_flag from <stdatomic.h>, which has been part of C since C11.

The compiler I use doesn't have atomics, and the compilers you refer to, do not support segmentation.

you don't need any tricks or 'fancy constructs' here. the aforementioned atomics work trivially and cleanly.

i'd go as far as to say that asm ("cli") is neither a hack nor unclean. it conveys a clear purpose. of course, you should hide such an architecture specific detail behind a layer of indirection. ditto for many other cases of inline assembly, especially extended inline assembly.

I'm more inclined to avoid ugly and unreadable portable code. In portable code, you never now what the code does or even if it's compiled or not due to the heavy use of conditional compilation. You also typically have layers of redefinitions so you don't even now what basic types are.

In my project, ifdef is forbidden, and so is constructing "portable" types & code.

Simply put, I don't care about portability. I only care about x86.

it is not particularly odd to write stubs in assembly. your original description makes it sound like the entirety of your irq code is in assembly, which'd be an utter waste of time.

My IRQs are pure assembly. A majority of the kernel code is assembly too. For example, the USB stack is in assembly, so is all network drivers, AHCI, IDE, APIC and a lot more.

[Octocontrabass]

When a new one is raised (because another character arrives), the old one is lost if the port has no FIFO. So, the issue is that with high enough baudrate, and no (or small) FIFO, characters will be lost if not handled fast enough.

I'm not sure serial ports can go fast enough for that to be a problem, as long as the driver cooperates with the scheduler. Only truly ancient serial ports lack a FIFO. Keyboards and mice are not fast enough for it to be a problem.

The compiler I use doesn't have atomics, and the compilers you refer to, do not support segmentation.

They do support segmentation.

Simply put, I don't care about portability. I only care about x86.

That's fine, but some of us want to run our OSes on more than just x86.

[ArsenArsen]

Spin locks can be implemented using atomic_flag from <stdatomic.h>, which has been part of C since C11.

The compiler I use doesn't have atomics, and the compilers you refer to, do not support segmentation.

gnu c supports named address spaces; though, to be fair, due to the useless nature of (edit)x86(/edit) segmentation for anything but tls, only __seg_{fs,gs} are implemented.

you don't need any tricks or 'fancy constructs' here. the aforementioned atomics work trivially and cleanly.

i'd go as far as to say that asm ("cli") is neither a hack nor unclean. it conveys a clear purpose. of course, you should hide such an architecture specific detail behind a layer of indirection. ditto for many other cases of inline assembly, especially extended inline assembly.

I'm more inclined to avoid ugly and unreadable portable code. In portable code, you never now what the code does or even if it's compiled or not due to the heavy use of conditional compilation. You also typically have layers of redefinitions so you don't even now what basic types are.

only small amounts of code necessarily need to be changed between architectures. a well-designed kernel shows no signs of having portability.

In my project, ifdef is forbidden, and so is constructing "portable" types & code.

Simply put, I don't care about portability. I only care about x86.

portability isn't the primary motivator behind writing readable code

it is not particularly odd to write stubs in assembly. your original description makes it sound like the entirety of your irq code is in assembly, which'd be an utter waste of time.

My IRQs are pure assembly. A majority of the kernel code is assembly too. For example, the USB stack is in assembly, so is all network drivers, AHCI, IDE, APIC and a lot more.

then the original comment applies

[rdos]

portability isn't the primary motivator behind writing readable code

It's more a kind of law that portability leads to unreadable code. At least, all the portable projects I've studied are a nightmare when you try to port them to something that is not Posix compatible. Often, it's not even possible to figure out which ifdefs are active and how basic types are defined unless you can build the project on Linux.

[ArsenArsen]

portability isn't the primary motivator behind writing readable code

It's more a kind of law that portability leads to unreadable code. At least, all the portable projects I've studied are a nightmare when you try to port them to something that is not Posix compatible. Often, it's not even possible to figure out which ifdefs are active and how basic types are defined unless you can build the project on Linux.

it is almost entirely unnecessary to use conditional compilation when writing portable code. some people doing so doesn't mean that everyone must. again, well written code will not show signs of portability.

as for non-posix platforms, there are many general solutions, such as cygwin or gnulib. either boils down to, again, implementing some subset of posix or some other api that your program uses on a layer below your program.

[bellezzasolo]

it is not particularly odd to write stubs in assembly. your original description makes it sound like the entirety of your irq code is in assembly, which'd be an utter waste of time.

My system is that by default everything gets routed through an interrupt manager, which handles things like the APIC EOI for you, as well as offering interrput number and a parameter block.

However, you're allowed to register native interrupts direct to the IDT for that CPU, if you really want.

Code: Select all

void register_native_irq(size_t vector, uint32_t processor, void* fn, void* param)
{
	arch_reserve_interrupt_range(vector, vector);
	IDTR current_idt;
	x64_sidt(&current_idt);
	IDT* idt = reinterpret_cast<IDT*>(current_idt.idtaddr);
	register_irq(&idt[vector], fn);
}

void register_native_postevt(size_t vector, uint32_t processor, void(*evt)())
{
	//Unsupported
}

static arch_interrupt_subsystem subsystem_native
{
	&register_native_irq,
	&register_native_postevt
};

void register_dispatch_irq(size_t vector, uint32_t processor, void* fn, void* param)
{
	uint32_t cpuid = pcpu_data.cpuid;
	if (processor != cpuid && processor != INTERRUPT_CURRENTCPU)
	{
		//Tell the AP to reserve the interrupt range
		//kprintf(u"Failed to register interrupt %x, CPU %d, running %d\n", vector, processor, cpuid);
	}
	else
	{
		arch_reserve_interrupt_range(vector, vector);
	}
	dispatch_tree*& disptree = dispatch_funcs[vector];
	if (!disptree)
		disptree = new dispatch_tree;
	dispatch_data* disp;
	if (disptree->find(cpuid) == disptree->end())
		disp = (*disptree)[cpuid] = new dispatch_data;
	else
		disp = (*disptree)[cpuid];
	disp->func = fn;
	disp->param = param;
	disp->post_event = nullptr;
}

void register_dispatch_postevt(size_t vector, uint32_t processor, void(*evt)())
{
	uint32_t cpuid = pcpu_data.cpuid;
	if (processor != cpuid && processor != INTERRUPT_CURRENTCPU)
		return;
	dispatch_tree*& disptree = dispatch_funcs[vector];
	if (!disptree)
		disptree = new dispatch_tree;
	(*disptree)[cpuid]->post_event = evt;
}

static arch_interrupt_subsystem subsystem_dispatch
{
	&register_dispatch_irq,
	&register_dispatch_postevt
};

[rdos]

So you have the IDT per CPU core? That's an interesting idea, but you would need some common exceptions to be shared.

Given that IRQs are a limited resource, and some modern devices want a lot of them, having the IDT per core might better support such devices. However, that means the IRQ setup code must know which core the IRQ is suppose to occur on, and the core of course must be running. I usually default all IRQs to BSP, and only start additional cores when load goes up. I then move both IRQs and server procedures to other cores when they use a lot of CPU time.

A possible strategy to keep IRQs dynamic is to allocate IRQs as private on BSP, and when they are moved to another core, deallocate on BSP and reallocate on the new core.

The Intel i210 network chip provides an interesting receive queue scheme where 5 different IRQs can be used, and in the optimal case they should be allocated on different cores.

[bellezzasolo]

So you have the IDT per CPU core? That's an interesting idea, but you would need some common exceptions to be shared.

Given that IRQs are a limited resource, and some modern devices want a lot of them, having the IDT per core might better support such devices. However, that means the IRQ setup code must know which core the IRQ is suppose to occur on, and the core of course must be running. I usually default all IRQs to BSP, and only start additional cores when load goes up. I then move both IRQs and server procedures to other cores when they use a lot of CPU time.

A possible strategy to keep IRQs dynamic is to allocate IRQs as private on BSP, and when they are moved to another core, deallocate on BSP and reallocate on the new core.

The Intel i210 network chip provides an interesting receive queue scheme where 5 different IRQs can be used, and in the optimal case they should be allocated on different cores.

Yeah, the crucial thing there was to support a per-core APIC timer, which is used for preemption. That said, the IRQ setup code doesn't need to know in most cases - all that is handled by the PCI MSI(-X) layer and interrupt dispatcher. Although that needs fleshing out, currently they're assigned to the CPU running the initialisation procedure, which is the BSP. The idea though is that the OS handles load balancing and such, all the driver has to do is assign a particular function and parameter to a particular PCI MSI. Generally the parameter is an opaque class pointer, and the function is a static that just casts the class and calls a member function to handle the interrupt.

How that interoperates with user applications requesting resources will be a little fun, but there we go.