ELF Questions

[MichaelFarthing]

I am working on the program loading mechanics for my OS and moving from a custom header to an ELF header. Currently the OS handles static executables only and so I am implementing ELF with a program Header but no sections and no Section Header Table (which is only really needed for dynamic linking). Though this allows me to load an executable image and works successfully there are some other things I am wanting to achieve and wonder if others can advise me how (for example) these sorts of things are achieved in Linux

However, without sections it seems difficult to identify the specific functions of particular segments. At present I am planning to use the operating system specific range of values for p_type to identify segment type to enable the program loader to do these tasks (which are done in my existing loader):

(1) Load code and some read-only data relative to the cs selector and other data (including a heap and stack null initialised area) relative to the other (identical) selectors. This enables further instances of the program to share the same code area but with different (write) data areas.

(2) Preset the stack pointer before entering the program. Maybe that should be a program responsibility but it seems a useful service to provide and would also enable debuggers to alert on detecting some stack problems (ie overflow).

(3) Enable my debugger to independently load the symbol data from the file image and orientate itself to the program structure (eg matching symbolic names relative to cs or ds/ss as appropriate)

I think my flag method will work quite satisfactorily but I feel uneasy in using custom flags for what looks like a common need. Can anyone advise me how (for example) Linux tackles these questions (or indeed other operating systems)? .

[bzt]

Currently the OS handles static executables only and so I am implementing ELF with a program Header but no sections and no Section Header Table (which is only really needed for dynamic linking).

Nope, you can have a PT_DYNAMIC record in the program headers to include all information needed for dynamic linking, meaning no sections required at all. Section header could be missing entirely.

However, without sections it seems difficult to identify the specific functions of particular segments.

Yes, that's the reason why in PHDRS block you can map sections into different segments, to help you with those identifications.

At present I am planning to use the operating system specific range of values for p_type to identify segment type to enable the program loader to do these tasks (which are done in my existing loader):

This is the expected way. You should be able to load ELF binaries without section headers (only program headers should count on execution).

(1) Load code and some read-only data relative to the cs selector and other data (including a heap and stack null initialised area) relative to the other (identical) selectors. This enables further instances of the program to share the same code area but with different (write) data areas.

Just put the read-only data sections to the same segment as the code and you'll be fine. You must have a different segment for the heap and the stack, since those cannot be read-only, and probably should be non-executable.

(2) Preset the stack pointer before entering the program. Maybe that should be a program responsibility but it seems a useful service to provide and would also enable debuggers to alert on detecting some stack problems (ie overflow).

Setting the initial stack pointer is always a task for the kernel, and cannot be the program's responsibility because the command line arguments of the invocation must be placed there, therefore the stack must be set up independently to the ELF executable. Kernel setting its own stack in _start is the exception, not the rule. Normal programs should assume they were provided with a valid stack on execution.

(3) Enable my debugger to independently load the symbol data from the file image and orientate itself to the program structure (eg matching symbolic names relative to cs or ds/ss as appropriate)

You won't have symbol data by default in an ELF executable (shstr, symstr and symtab are sections not segments). There's a bug in the GNU ld which prevents you to put those in a segment, however some symbols (depending on visibility) can be added to the dynamic segment if you're linking a shared library. FYI, debuggers usually use additional debugger information, and not the "basic" symbol table section, however that could be done for sure too (this is what gcc's "-g" flag is for).

I think my flag method will work quite satisfactorily but I feel uneasy in using custom flags for what looks like a common need. Can anyone advise me how (for example) Linux tackles these questions (or indeed other operating systems)? .

They usually don't care about sections, require a dynamic segment, and propagate the linking to an "interpreter" (which is independent to the kernel, is a separate user-space library). Sections are only used for compile-time static linking, but not for execution (neither for run-time linking).

Hope these help.

Cheers,
bzt

[nullplan]

(1) Load code and some read-only data relative to the cs selector and other data (including a heap and stack null initialised area) relative to the other (identical) selectors. This enables further instances of the program to share the same code area but with different (write) data areas.

You do not need to know this. There is a bunch of PT_LOAD segments in the file, and you only need to map those into address space. Which of them is code, which is data, and which is something else entirely is something you don't have to know. You set the protection attributes according to the p_flags in the program header and that is all you need to do. Bear in mind the difference between ET_EXEC and ET_DYN (the former having base address 0 and the latter having a random one aligned to the highest alignment among the PT_LOAD segments), and you should be golden.

(2) Preset the stack pointer before entering the program. Maybe that should be a program responsibility but it seems a useful service to provide and would also enable debuggers to alert on detecting some stack problems (ie overflow).

Yeah, that is always a bit of a mess. You need to allocate "enough" memory for the stack, with no indication how much is "enough", need to map the thing somewhere were it does not conflict with the other mappings, need to expand it when the application runs out... complicated.I would recommend mapping a guard page below the stack. If a page fault occurs in the guard page, expand the stack if possible, and kill the process if not. That doesn't help for where to map the stack, though. For compatibility reasons, many OSes will use a place high-up in address space (as far up as possible in user space), though Linux mixes some randomness in, and then it gets fun.

(3) Enable my debugger to independently load the symbol data from the file image and orientate itself to the program structure (eg matching symbolic names relative to cs or ds/ss as appropriate)

Symbol table is extra. Debugging data is even more extra. Both of these are in sections, and debugger data is very complicated (look up the DWARF spec if you are interested).

[Octocontrabass]

(1) Load code and some read-only data relative to the cs selector and other data (including a heap and stack null initialised area) relative to the other (identical) selectors. This enables further instances of the program to share the same code area but with different (write) data areas.

ELF segments are unrelated to x86 segments. Read-only data is shared by mapping the same read-only pages into multiple address spaces. The CS, DS, ES, and SS bases should all be 0, and the limits should all be 4GiB-1. Paging provides all R/W/X protection. (Unless you're targeting ancient CPUs without the NX bit.)

Setting the initial stack pointer is always a task for the kernel, and cannot be the program's responsibility because the command line arguments of the invocation must be placed there, therefore the stack must be set up independently to the ELF executable.

It's a good idea to have your OS set up the initial stack and put the command line arguments on it, but you don't need to do it that way if you're not aiming for binary compatibility with another OS. You can always do it differently if you think of something better.

[MichaelFarthing]

Thanks all for these detailed replies which has helped my thinking considerably. My main worry was to ensure I kept reasonable compatibility with other OSes. Its nice to be back here after a long absence. I left partly because at the time the forum had got itself wrapped up with an existential quarrel but also because I was too gripped by what I was doing myself to want to spare time! (particularly as at that point my effort was going into my own compiler rather than the OS itself). I'm sure flames still happen, but this supportive welcome back shows how good this forum can be.

Incidentally, I already have my own functioning debugger (though a bit crude). The intention is that the symbolic data exists in its own segment of the file image and is not loaded by the program loader. Nevertheless, as the elf header itself will be loaded to the image a debugger would be able to access this and thus independently load the symbolic data. [Currently the debugger depends on the symbol data always being loaded as part of the executing image].

[nullplan]

I'm sure flames still happen, but this supportive welcome back shows how good this forum can be.

Gee, thanks. It helps that you ask your questions well. I find that I modulate the effort that goes into an answer by the effort that is evident in the question. If somebody, like you, shows their work in the question given, I have no qualms about spending some time formulating an answer. If, on the other hand, the question betrays not just a lack of knowledge, but utter will to do the most basic steps to troubleshoot or investigate, then suddenly all motivation leaves me, and they get a terse answer at best.

The intention is that the symbolic data exists in its own segment of the file image and is not loaded by the program loader. Nevertheless, as the elf header itself will be loaded to the image a debugger would be able to access this and thus independently load the symbolic data. [Currently the debugger depends on the symbol data always being loaded as part of the executing image].

That is pretty much exactly what is happening with most ELF debuggers. If an ELF file has symbols, it will have a section header table (given in the ELF header). In that section header table, there may be a section of sh_type SHT_SYMTAB. That section header will contain an sh_link field pointing you to a string table (section of type SHT_STRTAB). When parsing the symbol table, each symbol contains a field "st_name", which is an offset into that linked string table where you can find the symbol name.

There are a few other things to know about symbol tables, but the point is, they will not be part of any LOAD section, and therefore not available to a program only reading the memory mappings.