Higher Half Kernel necessary for my use case?

[LyricalRain]

I'm developing for UEFI, x86-64. Considering this is but something i do as a hobby and to learn, I do not see myself releasing this OS for others to use.

Now, onto the problem. It's been a pain to map a higher half kernel using UEFI. Maybe I'm not intelligent enough, but all solutions I've tried have not worked. I tried mapping to a higher half in the "efi_main" function before my kernel is called. UEFI did not let me modify its paging tables, and any attempt to change the CR3 register to point to my own PML4 table was met with triple faults as well. It could be done in the kernel itself but until that point, I'll either have to use offsetted addresses every time or split the kernel into 2 and link them separately, which I do not know how to do.

So I've been thinking, is it really necessary to have a higher half kernel, for my specific use cases? As I'm the only one who will use this OS, I could very well statically link every user program to where I know nothing is present. Is this a fine thing to do? Because I'd rather move on to actual kernel development before I waste too much time on this. What does the community think?

[Octocontrabass]

UEFI did not let me modify its paging tables, and any attempt to change the CR3 register to point to my own PML4 table was met with triple faults as well.

The way UEFI is designed, it's probably easiest to have a separate bootloader that will load your kernel binary from the disk, then exit boot services, then set up initial page tables to put the kernel at its desired virtual address, then call the kernel.

What does the community think?

UEFI doesn't guarantee available memory at any particular physical address, so you can't have a kernel linked at a fixed address that will run on every UEFI PC.

But it's your OS, and you can do what you want with it.

[bzt]

UEFI did not let me modify its paging tables, and any attempt to change the CR3 register to point to my own PML4 table was met with triple faults as well.

You have to call ExitBootServices first. After that you can do whatever you want.

It could be done in the kernel itself but until that point, I'll either have to use offsetted addresses every time or split the kernel into 2 and link them separately, which I do not know how to do.

No need. Set up the paging table in your UEFI loader, and link your kernel at the desired virtual address, no need to split it.

So I've been thinking, is it really necessary to have a higher half kernel, for my specific use cases? As I'm the only one who will use this OS, I could very well statically link every user program to where I know nothing is present. Is this a fine thing to do?

You've told us nothing about the specific use case. Boot loader and static linking has almost nothing to do with higher-half. If you want your processes to have separate address spaces, that's when you have to map the kernel in each and every address space, and practically that should be at a higher address.

Because I'd rather move on to actual kernel development before I waste too much time on this. What does the community think?

Give a try to BOOTBOOT. It will load and boot your higher-half kernel on UEFI (and on other platforms too), you don't have to waste time on the loader, you can start writing your kernel right away. As a bonus, it comes with a small, dependency-free mkbootimg utility to create a bootable disk image, and it will also set up the video framebuffer for you.

Cheers,
bzt

[nullplan]

UEFI did not let me modify its paging tables, and any attempt to change the CR3 register to point to my own PML4 table was met with triple faults as well.

That means you made a mistake. Common ones are forgetting to call ExitBootServices() before attempting that, forgetting the identity map for the change over, or, of course, getting the page table format wrong. Since you provided no code and my glass ball is currently in the wash, I can only speculate here.

The identity map is necessary, because whenever you change CR3, the current code address must map to the same physical address before and after the change. And UEFI only has an identity map going. ExitBootServices() is also necessary, because the boot services might have interrupt handlers registered, and those might depend on the old memory map. Mind you, if your PML4 includes a proper identity map for all physical RAM, then those interrupts should not be able to tell the difference.

So I've been thinking, is it really necessary to have a higher half kernel, for my specific use cases?

Necessary? No. But I do recognize avoidance behavior when I see it. This design choice of yours is not based on some engineering principle, but rather on frustration. The proper response to frustration is to sit down and learn. No, a higher half kernel is not necessary for you, but getting the page tables right is actually one of the easier exercises in OS development. I guarantee you it is not going to get easier. If you already fall at the page tables, how will you deal with the frustrations of building a memory management unit? Building a graphics driver? Actually, drivers in general. You do something wrong there, and the hardware just folds its arms and refuses to cooperate. Are you going to ask us then if having a network card or USB support or whatever is strictly necessary for your use case?

There is also the practical aspect Octo mentioned: Since no address is guaranteed to be free, but you wish to remain identity mapped, your kernel would have to be position independent. This is possible but requires loader support. A higher half kernel might actually be simpler here, since it is just a normal statically linked executable.

[LyricalRain]

Thank you for your replies everyone. I do realize that it was just avoidance behaviour and I'll try to get it working. Probably made a mistake in my paging tables themselves somewhere.

[LyricalRain]

Give a try to BOOTBOOT. It will load and boot your higher-half kernel on UEFI (and on other platforms too), you don't have to waste time on the loader, you can start writing your kernel right away. As a bonus, it comes with a small, dependency-free mkbootimg utility to create a bootable disk image, and it will also set up the video framebuffer for you.

I actually tried using BOOTBOOT to surpass bootloading altogether and get directly to the kernel. I got it working too. However, no matter what resolution I set in the config file, Qemu always runs at 640x480, and the resolution in the bootboot structure also says 640x480. And I know it's not a problem about detecting the config file because the kernel directory field works as expected. Any idea what might be causing the issue?

[bzt]

I actually tried using BOOTBOOT to surpass bootloading altogether and get directly to the kernel. I got it working too. However, no matter what resolution I set in the config file, Qemu always runs at 640x480, and the resolution in the bootboot structure also says 640x480. [...] Any idea what might be causing the issue?

Have you recreated the disk image? The config file in your source repo has no use when you run qemu, only the config file inside the disk image matters.

Other than that, if BOOTBOOT can't find the config file, it sets up the native resolution of the display, or if that can't be found either, then 1024 x 768. This suggests either you have an (un-updated) config in the disk image that tells to set up 640 x 480, or you have configured qemu with a very few RAM, so there's not enough memory for bigger than 640 x 480 resolutions. I had issues with certain Tianocore versions that it couldn't run properly with default qemu configuration, I had to specify "-m 64" (at least 64 MB RAM, but 128 more likely) on the qemu's command line to avoid these issues. You can also try to use different VGA ROMs, see "-vga" option, I use cirrus and it works for me.

To locate the problem, first comment out the "screen" in the config file and recreate the disk image. If it still sets up 640 x 480, then the problem is in your qemu configuration (or Tianocore is explicitly told to use 640 x 480 as native resolution). If it sets up 1024 x 768, then you forgot to update the disk image with the new config file.

If everything else fails, set GOP_DEBUG define to 1, and recompile (only standard GNU toolchain needed). With that it will dump all the available modes that Tianocore reports, with the native resolution marked by an asterisk, so you'll be able to see if it reports higher than 640 x 480 resolutions at all (use "-serial stdio" on qemu's command line to see the debug messages). Alternatively use the same qemu configuration, but replace EFI/BOOT/BOOTX64.EFI in the disk image with this. It should list the available modes too, and no BOOTBOOT recompilation necessary.

Hope this helps.

Cheers,
bzt

[LyricalRain]

Other than that, if BOOTBOOT can't find the config file, it sets up the native resolution of the display, or if that can't be found either, then 1024 x 768. This suggests either you have an (un-updated) config in the disk image that tells to set up 640 x 480, or you have configured qemu with a very few RAM, so there's not enough memory for bigger than 640 x 480 resolutions. I had issues with certain Tianocore versions that it couldn't run properly with default qemu configuration, I had to specify "-m 64" (at least 64 MB RAM, but 128 more likely) on the qemu's command line to avoid these issues. You can also try to use different VGA ROMs, see "-vga" option, I use cirrus and it works for me.

To locate the problem, first comment out the "screen" in the config file and recreate the disk image. If it still sets up 640 x 480, then the problem is in your qemu configuration (or Tianocore is explicitly told to use 640 x 480 as native resolution). If it sets up 1024 x 768, then you forgot to update the disk image with the new config file.

If everything else fails, set GOP_DEBUG define to 1, and recompile (only standard GNU toolchain needed). With that it will dump all the available modes that Tianocore reports, with the native resolution marked by an asterisk, so you'll be able to see if it reports higher than 640 x 480 resolutions at all (use "-serial stdio" on qemu's command line to see the debug messages). Alternatively use the same qemu configuration, but replace EFI/BOOT/BOOTX64.EFI in the disk image with this. It should list the available modes too, and no BOOTBOOT recompilation necessary.

Hope this helps.

Cheers,
bzt

Yup it was the OVMF.fd file. I used another one and now it works properly. Thanks!