Announcing POSIX-UEFI

[bzt]

Yeah. Maybe these might shed some light on the topic, both sentences from Wikipedia:

The POSIX specifications for Unix-like operating systems originally consisted of a single document for the core programming interface

Before 1997, POSIX comprised several standards: POSIX.1: Core Services (incorporates Standard ANSI C) (IEEE Std 1003.1-1988)

Cheers,
bzt

[bzt]

Hi All,

I've added UTF-8 support. This means no more fiddling with wchar_t and L"" literals any more:

Code: Select all

#include <uefi.h>

int main (int argc, char **argv)
{
  printf("Hello, world!\n");
  return 0;
}

Just as with any other POSIX systems (except for the includes)! You can still get the old behaviour (main(int, wchar_t) and printf(L"")) if you comment out USE_UTF8 in uefi.h. This will make POSIX-UEFI less POSIXish, but there will be no string conversions between your application and the UEFI interfaces, all strings will be passed as-is. To write code for both modes, char_t is defined (either char or wchar_t), and the CL() macro takes care of the L prefix for constant literals.

I've also made LLVM the default toolchain, from now on you can switch to gcc by using USE_GCC in the Makefile.

Cheers,
bzt

[nullplan]

I've added UTF-8 support.

Cool. As a proponent of "UTF-8 everywhere", I think that is a positive development.

[Korona]

Agreed, UTF-16 is quite tedious to work with.

[bzt]

Thanks! I'm an UTF-8 fan too. Plus this allowed me to use "char" in the API, so now POSIX-UEFI really looks like the usual ANSI C libc interface. Except getenv and setenv, I could not help those because UEFI environment variables are binary blobs.

The other difference I'm thinking what to do with is fread. If you open a standard file with fopen, then you can use fseek and fread as usual. However if you open "/dev/diskX", then (because UEFI doesn't manage offsets for Block IO internally), fseek is useless and fread looks like "fread(buffer, size, lba, stream)", so instead of the number of blocks argument you pass the required lba address. I'm not sure what to do with this, as it is simple enough, and I'm not sure it worth implementing fseek for Block IO protocol. What is your opinion? BTW, I've added typedef structs for the GPT too.

Cheers,
bzt

[nexos]

I've added UTF-8 support.

That was my biggest bone with UEFI was that using a common codebase for legacy and EFI was tedious, as I had to have Unicode and ANSI functions separate throughout. I created some ugly macros that helped somewhat, but that's a great improvement, just using UTF-8 from the start!

[bzt]

Okay, I've added more examples:

• the mandatory Hello World of course
• getting the command line arguments
• dumping memory (feature of POSIX-UEFI's printf %D)
• listing directories (using opendir / readdir / closedir)
• Reading files (with fopen / fread / fclose)
• Dumping sectors from block devices (with fopen / fread / fclose), oh I've implemented the offsets for block devices btw
• Using serial port (with fopen / fwrite / fprintf / fread / fclose)
• Using GOP to query and set video modes (using UEFI API)
• Printing UTF-8 string with bitmap as well as vector fonts (with and without the SSFN library)

There are screenshots too, for example


I had to add that last UTF-8 printing part because I've tried to do

Code: Select all

ST->ConOut->OutputString(L"Здравствуйте\r\n");

and it doesn't work with TianoCore! WTF? Why the f*ck does UEFI mandate wchar_t then if it can't support non-Latin alphabets at all? Yet another indicator of M$ / Intel complete incompetence. Let's waste double space for every string without providing actual support for UNICODE BMP! Yeah, what could possibly go wrong? UEFI Consortium don't pay for it, so let's just waste the user's money, storage, RAM and network bandwidth! Grrr. Anyway, using my own fonts I was able to display UTF-8 text properly, so that's the preferred (and only) way of printing multi-lingual strings with POSIX-UEFI.



Cheers,
bzt

[Octocontrabass]

I had to add that last UTF-8 printing part because I've tried to do

Code: Select all

ST->ConOut->OutputString(L"Здравствуйте\r\n");

and it doesn't work with TianoCore! WTF? Why the f*ck does UEFI mandate wchar_t then if it can't support non-Latin alphabets at all?

It does support non-Latin alphabets, you just need to include your own font and register it with the HII database.

[bzt]

It does support non-Latin alphabets, you just need to include your own font and register it with the HII database.

Only in theory. UEFI Driver Writer's Guide says about HII fonts

The system carries the Latin-1 (Western European) character set. Other characters must be provided if they are to be displayed.

The same document also classifies "HII Font" a rarely used protocol, and does not care about providing a detailed description or an example on how actually do that, eg. how to "provide the characters to be displayed". Plus including the glyphs for the entire CJK area in each and every driver which needs Chinese translation as UEFI blogspot suggests is ridiculous.

Let's say I'm designing a plug-in card and I want to support Chinese. But I don't know whether or not the system firmware in the platform where my card is installed supports all the Chinese characters (glyphs) that I need. What can I do? Well, with UEFI 2.10, your driver can just carry the characters you need that are in addition to those provided by the ISO Latin-1 characters that the platform is required to carry in its firmware.

That's the way to do it! Yeah, let's duplicate 65536 - 256 glyphs in each and every driver! The firmware isn't bloated enough already! (sarcasm)


That's about theory. Now in practice, taking a deep dive in TianoCore's source, GetGlyphBuffer only supports SimpleFont with EFI_NARROW_GLYPH and hardcoded EFI_GLYPH_WIDTH. Forget that you set the font width from the font. Again, in theory GetGlyphBuffer consults GlobalFont->FontPackage, but there's no way to register sane fonts with it, as GraphicsConsole.c itself has hardcoded EFI_NARROW_GLYPH, and "This->Mode->CursorColumn * EFI_GLYPH_WIDTH" is all over the place. In RegisterFontPackage at line 2064 it only sets up package size for narrow fonts, does not check its input (it is a notification callback) which makes you wonder what a nice buffer overflow could happen if you actually try to register a wide font....

So the statement that UEFI supports proportional fonts is just yet another big lie. It doesn't. Even if you figure out how to register HII Font Packages, you cannot display glyphs with different sizes using OutputString, that's for sure.

Now why do both Font.c and GraphicsConsole.c define exactly the same palette over and over again but separately is beyond my comprehention (specially when they should not need it as colours are taken care of in StringToImage() in Image.c). When I first tried to understand TianoCore's source, I was just confused. Then I did so much FacePalms (tm), that my head became red Now I just laugh when I think about how hilariously overengineered and completely useless garbage UEFi is! And the more time I spent with it, the more convinced I became If someone says that UEFI is a progress over BIOS, then a) they are paid to say that b) they are simply fools or idiots.

Do you want to hear another epic fail? UEFI Firmware must keep track of allocated memory pool sizes, for BS->FreePool only receives a pointer. But so must YOUR APPLICATION TOO, if you ever want to reallocate a pool! Even though AllocatePool receives a memory pointer, it uses that for output only, so no reallocation possible, you must do that yourself in your application with memcpy! Which means you must keep track of the allocated memory pointers and sizes. You simply cannot not wonder, what a total idiot designed this interface?

Anyway, POSIX-UEFI hides all this sh*t, so that you can simply use stb_image.h to load PNG images for example:


Cheers,
bzt

[bzt]

Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Now the examples are complete, they cover all the topics one might need for writing their own boot loader: loading files or sectors, printing UTF-8 strings, sending formatted messages to serial for debugging, getting memory map, setting video modes, displaying PNG images, loading and executing ELFs, passing boot parameters to it, exit Boot Services etc. Everything is self-contained and dependency-free, and I've tried to write all code to be small and easy to follow. Longest being about 130 SLoC.

https://gitlab.com/bztsrc/posix-uefi/-/ ... r/examples

Cheers,
bzt

[thewrongchristian]

Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Now the examples are complete, they cover all the topics one might need for writing their own boot loader: loading files or sectors, printing UTF-8 strings, sending formatted messages to serial for debugging, getting memory map, setting video modes, displaying PNG images, loading and executing ELFs, passing boot parameters to it, exit Boot Services etc. Everything is self-contained and dependency-free, and I've tried to write all code to be small and easy to follow. Longest being about 130 SLoC.

https://gitlab.com/bztsrc/posix-uefi/-/ ... r/examples

Cheers,
bzt

Bravo, good work!

[Octocontrabass]

Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Do you have any more details about the bug or your workaround? I'd like to know more.

[bzt]

Added more examples, plus I've workarounded an extremely annoying GCC bug with incorrectly inlining builtins, even with -fno-builtin...

Do you have any more details about the bug or your workaround? I'd like to know more.

Sure.

TL;DR GCC incorrectly inlined memcmp() under some circumstances, which resulted in an infinite loop.

I noticed it in fopen line 215. This line

Code: Select all

if(!memcmp(__filename, CL("/dev/serial"), 11 * sizeof(char_t))) {
  printf("match\n");
  ...
}
printf("no match\n");

froze in an infinite loop with the memcmp call, and nothing was printed. The first argument is checked not to be null, the second argument is a constant string literal (CL() macro does nothing), and the third is again a constant, not zero. There is no UB here, and memcmp() shouldn't freeze for sure. Taking a look at objdump -d stdio.o and no wonder that the code froze, the inlining was completely wrong. Interestingly it worked from the user app (where it was compiled as a call to memcmp@PLT), but not in the other object files of the library, like stdio.o or stdlib.o. It also worked with CLang as it should. I've used a naming trick to prefix these problematic functions with the C-precompiler, so that GCC won't try to replace my memcmp with its own buggy one, and problem solved. Now memcmp is called "memcmp" with Clang, and "__uefi_memcmp" with GCC, in which case a define makes it possible that user app can use "memcmp" as usual.

Cheers,
bzt

[kzinti]

The optimizer simply detected that it could optimize your code by calling... memcpy(), resulting in infinite recursion. The same thing can happen with memset().

You can work around this by disabling the specific optimization causing the problem. Here is what I did for my memset():

Code: Select all

#pragma GCC optimize "no-tree-loop-distribute-patterns"

I have also run into a very similar problem where GCC optimizes malloc() + memset() into calloc(). The problem was that I was providing my own implementation of calloc() that was basically doing malloc() + memset(), resulting in another infinite loop. This was was fixed by:

Code: Select all

#pragma GCC optimize "no-optimize-strlen"

[bzt]

The optimizer simply detected that it could optimize your code by calling... memcpy()

Sadly no, that would make some sense at least. There was no call to "memcpy" at all, instead just an incorrect test eax, eax loop.

You can work around this by disabling the specific optimization causing the problem.

Which weren't enabled in the first place Plus I have no control over what optimizations the user wants to use. Using the __uefi_ prefix works, as GCC doesn't think it's memcmp any more

The problem was that I was providing my own implementation of calloc() that was basically doing malloc() + memset(), resulting in another infinite loop. This was was fixed by:

Code: Select all

#pragma GCC optimize "no-optimize-strlen"

Eeeeeek.... What has strlen to do with that? But yeah, I know this bug, I've also run into that. And another, if you have a struct at address 0, then accessing any of its fields (at non-zero address) would generate an UD2 instruction instead of a MOV, how lovely, isn't it?

Cheers,
bzt