Desire to make my own executable format, like ELF, COFF, PE?

[LieutenantHacker]

I have looked up a little bit on the executable and linkable format, common object file format, etc.

I decided that for my own hypothetical kernel I wish to make one day I want my own executable format programs.

I will not mention any specifications, extensions, etc. to avoid thievery and such, but what would be the general steps of doing this?

In other words, what denotes any applicable difference of executable formats, and how do they have an advantage from just a flat binary itself?

[dozniak]

You go on about it by implementing some part of the toolchain for your needs. There are very many ways to achieve it - you may write your own linker, you may write an output format handler for nasm, ld or objcopy, you may write a tool to grab certain standard format file and some metadata and output your own - that's pretty much like your own linker, but you avoid doing the relocation calculation and all other nitty-gritty and only do reshuffling of output sections to your own liking.

After all that I recommend you simply use ELF. You even can extend it with your own section types if you need to; it's a very well supported, portable and flexible format.

[Brendan]

Hi,

I will not mention any specifications, extensions, etc. to avoid thievery and such, but what would be the general steps of doing this?

In other words, what denotes any applicable difference of executable formats, and how do they have an advantage from just a flat binary itself?

First determine what you need from an executable file format. Features may or may not include:

• some sort of identifiers; so you can determine if the executable is for (e.g.) 32-bit 80x86 or 64-bit 80x86 or ARM or something else (e.g. some sort of portable byte-code)
• flags to tell the OS what the executable needs - e.g. if it needs special privileges (root access) or special features (e.g. SSE) or should be treated differently by the OS (e.g. device driver and not a normal process)
• some sort of correctness check - e.g. a CRC so that the OS can detect if the executable file was corrupted (or maybe some sort of digital signature)
• some way to split the executable into "sections", where different sections have:
  • a virtual address to say where the section belongs in the virtual address space
  • some attributes - e.g. if the section is in the file or if its like the ".bss" where there's no data in the file, or if the section contains special information that isn't loaded into the virtual address space
  • permission flags (e.g. executable, readable, writeable) that the OS can enforce (e.g. so that if there's a bug that causes a write to a "read only" section you get a page fault)
• support for relocatable code (e.g. position independent shared libraries that are happy to be at any virtual address)
• support for dynamic linking
• support for multiple targets ("fat binaries") where the same executable might include sections for 2 or more different platforms (e.g. where the sections for 32-bit 80x86 are used on 32-bit 80x86 machines and the sections for PowerPC are used on PowerPC machines)
• debugging information; so that software (e.g. a debugger) can figure out which pieces of executable code came from which lines of source code and/or so you can find the names of functions, variables, structures, etc.
• metadata; so that software can determine things like what sort of copyright/license the executable uses, who wrote it, where it came from, which version it is, etc. Note that metadata could also be used for some sort of "auto-update" feature (e.g. where the OS automatically checks a URL for more recent versions), or for automated bug reporting (e.g. where the OS sends a "your executable crashed" email to the developers containing anything that might help them fix the bug), or a list of keywords (e.g. so the user can search for "word-processing" and the OS can find all executables that have the "word-processing" keyword in their metadata)
• an icon (for GUIs); so that "foo.exe" is represented by a little picture of a foo
• anything else you can think of

Once you've figured out what you want; I'd strongly recommend writing some sort of specification that describes the file format. Even if you only use it yourself (and have no intention of letting anyone else see it), it makes it a lot easier to get the design right (and can help to avoid lots of mistakes) and easier to figure out what you've done. You'll end up referring to your specification a lot while you're writing code to support the executable file format (e.g. linker/compiler, debugger, loader, etc).


Cheers,

Brendan

[Love4Boobies]

I want to raise the question whether you've given this secret executable file format specification enough thought. If you keep it a secret, this basically means that implementers cannot develop compilers for your platform without going through the wasteful and error-prone process of reverse engineering your own compiler and/or loader. I do not understand the trade-off you are trying to make; what is there to be gained?

[iansjack]

In other words, what denotes any applicable difference of executable formats, and how do they have an advantage from just a flat binary itself?

if you don't know the answer to that, why do you want to produce your own executable file format? What's in it for you?

[Love4Boobies]

Learning, perhaps.

[iansjack]

I'd suggest that learning might be better served by studying existing executable formats, seeing what the differences between them are, trying to understan what they do - and how they do it - and then perhaps thinking about how one could improve on them. A vague idea to implement your own (anything) without really understanding what that anything does is likely to be a frustrating and non-productive exercise.

[Mikemk]

Learning, perhaps.

Or perhaps he wants to be the only one who knows the specification so that only his company (Or perhaps army?) can write programs for it.

[sortie]

I considered doing my own executable file format when I started out. Originally, I figured I'd start with a.out or something custom. After looking into ELF I spent a couple hours writing a ELF parser and I had support for it, compared to the weeks (months?) of work retargetting my toolchain to understand the new format. Coming up with a new executable format is easy, adding support for it to existing software not-so-much.

[Mikemk]

I considered doing my own executable file format when I started out. Originally, I figured I'd start with a.out or something custom. After looking into ELF I spent a couple hours writing a ELF parser and I had support for it, compared to the weeks (months?) of work retargetting my toolchain to understand the new format. Coming up with a new executable format is easy, adding support for it to existing software not-so-much.

I'm currently facing this problem for a custom filesystem, and may abandon it and implement a normal fs now, custom later.

[AbstractYouShudNow]

Or perhaps he wants to be the only one who knows the specification so that only his company (Or perhaps army?) can write programs for it.

Army ? Damn! I didn't know we were preparing to a war XD

As I always say, formats are just a detail when the design is good enough. Essentially, you can just support more than one executable format, and so there's no need to develop your own, just exploit an existant one, or use the customizing capabilities of a free one (like ELF).

But before you consider creating your own, I'll follow the people and tell you that you should first do research about this kind of thing. Start on Google/Bing/<anything>Search, Wikipedia, and other such common informations sources. Take notes about everything that seems important. Then, once you gathered all the information, add your own requirements, and correlate (put in relation) all these informations. Then, study some early design documents of an existing (and popular ?) format.

The other thing you should consider is if it's really worth it. You must know (and I speak as both an osdev and hosted applications dev) that the world of computers is made more and more complicated and unmanageable by too many people wanting to do their own format. Take the case of image formats: there are thousands of formats that basically do the same thing, just because a guy wanted to do his own format, and besides it, a company wanted to have their own proprietary format (Adobe's psd files), and another group wanted to make a format that's more free, and still another group thought they could do more serious work, and so it goes on and on... What you should understand is that designing your own executable format is worth the effort only if you can add a significant progress to the current technologies, otherwise, you're better off tweaking ELF's custom sections or the like. Otherwise, you will have a hard time integrating your format into existing tools, and even to get yourself supporting your own format and compiling applications in it. Moreover, if you eventually want that some day people create applications for your OS, they will be stopped by this additionnal difficulty (learning a new set of APIs is already a big one).

So for your researches, I think you can start up with what Brendan told you about what an executable format must do, I'm pretty sure he's right (I never saw him be wrong, anyways), but you need to do further researches before diving in the implementation. Otherwise, you'll end up like an hypothetical God trying to create a dog, only knowing that it must shout and bite; his creation could be any kind of beast, but the probability is small that he will actually end up with a dog

[rdos]

It depends if the executable format is for kernel (device drivers) or for applications.

Doing your own executable format for applications, or any other user-level code is generally a bad idea. Most of the features of ELF or PE are needed, and it makes no sense to redesign those in some other ways. When using PE, it is possible to use some header-fields so that the executables doesn't load on Windows, and so that you know that Windows executables will not run on your OS. That's much easier to change than to roll your own format.

Defining your own device-driver executable format makes more sense. Most of the complexities of applications that are built-into both PE and ELF formats really aren't relevant for a device-driver. I've created my own 16-bit and 32-bit executable device-driver formats, which really only contain a small header and pre-relocated code. These are segmented, so I let the linker do all relocations, which include putting the correct code and data selector into the code. These selectors are then also put in the header so the loader knows which selectors to load when doing the initialization. The header also contains the sizes for the code and data segments.

A flat device driver needs to operate in a slightly different way, possibly leaving relocation until run-time, unless it is put at a particular address, or uses relocatable x86-64 code.

[LieutenantHacker]

I had originally thought of making my own executable format for both system software and application software.

Besides, if developing on my GUI/OS you would need to know the framework/APIs anyways, and the format would be handled by the compilation process anyways(no developer really needs to know that in order to develop).

Do most people who are developing application software on Windows worry about PE, etc.?

I also meant that I didn't want to share my ideas before they were put in to a specification; that way, nobody could jump the gun and develop off of my uncompleted idea(forking).

I really doubt adding a new executable format to a new OS hardly makes anything more difficult to the developer, since that's likely abstracted away from them by either a HLL, or the compiler and linker's job deals with that.

And to note, not all image file formats are the same and do the same thing. Someone's own implementation of an image format (like PNG to GIF) are not the same.

But good heads up for all of the information anyways!

[Combuster]

Do most people who are developing application software on Windows worry about PE, etc.?

For most windows developer it's like not knowing any better. The moment you start developing for multiple platforms, you will feel the force of the (unnamed) differences pretty soon, for better or for worse.

Someone's own implementation of an image format (like PNG to GIF) are not the same.

Inventing your own implies nobody will have her favorite tool available for the job, only yours which will be observed as inferior due to human nature.

If you're actually expecting others to work with it, it's not a good plan to make them suffer your NIH syndrome unless there's a practical reason why the existing tools are insufficient. In all other cases, feel free to entertain yourself.

[bluemoon]

It's good _if_ you got idea that even guys at microsoft (for PE), the SysV guys who designed ELF, and many other genius over-looked. Good luck.

However, if you mean special features linked with your OS, I would say it has nothing to do with the container format, and you can just add custom fields onto ELF, or zip multiple ELF together to implement multi-target.