Could Memory-Managment seem transparent to the programs?

[0x57]

If I make a Segmented memory in Real Mode, but later change it to a Paged one in Long Mode, I have couple of options like switching to the compatibility mode to run the old applications, right? I'm wondering is there any chance to make the memory management mechanism totally transparent to the applications, so they could run without modifications/recompile and without changing the processor mode, which means your Real Mode application would run in Protected or Long Mode and vice versa.

Sounds Stupid-Impossible question, however I appreciate any ideas or inspirations, thanks! :)

P.S. I was thinking of adding a proxy between the application and memory manager. It could provides an API for the program so you can add kinda of meta-data to the executable files' header section, so the proxy would do the proper modifications to the binary, and then try to execute it.

P.P.S. I know that we do the similar thing in Relocation Table, however I think this one needs more modifications on the binary ...

[iansjack]

Well, I'd say that memory management should always be a black box as far as applications are concerned. You call malloc() when you want memory, free() when you need it no longer (or you use some sort of garbage collection), and care not a fig how that is implemented. But a recompile will almost always be needed if you change the underlying memory model. I see no problem with that.

[qw]

I'm wondering is there any chance to make the memory management mechanism totally transparent to the applications, so they could run without modifications/recompile and without changing the processor mode, which means your Real Mode application would run in Protected or Long Mode and vice versa.

This is not just a matter of memory management. Machine code may have different meanings in different modes. For example, B8 34 12 BB 78 56 means in real mode, protected mode and long mode respectively:

Code: Select all

mov	ax, 0x1234
mov	bx, 0x5678

Code: Select all

mov	eax, 0x78bb1234
push	esi

Code: Select all

mov	eax, 0x78bb1234
push	rsi

[0x57]

But a recompile will almost always be needed if you change the underlying memory model. I see no problem with that.

Thanks iansjack, but two questions:

First, in which cases you mean a recompile will be needed? Could you give me a clear example, if you have any in mind.

Second, I thought that none of the major operating systems employing that, because (for example) you need an emulator to run a 16-bit program on 32-bit system. Do you know anywhere which there would be a documented stuff about using this technique in any operating system?

[0x57]

Thanks Hobbes ... well, something like a virtual machine and byte-code technique (e.g. java) would works then, right? just slower ...

[bluemoon]

You can subdivide "memory allocation" into different implementation layers:

KERNEL SIDE
1. page allocator (aka physical memory manager)
2. logical memory manager (this handle page mapping, etc)
3. sbrk() or HeapAlloc() alike API that enable application to request memory.

APPLICATION SIDE
4. malloc()/free() that manage memory pool allocated with sbrk/HeapAlloc; you may have multiple version of malloc.

Usually you don't need to rebuild application if you altered implementation on kernel side and keep the interface unchanged.
However, changing from 16-bit segmented to 64-bit basically require different pointer representation, ie. interface changes; and recompile is almost unavoidable if you want to actually make use of the extra bits.

[0x57]

First, in which cases you mean a recompile will be needed? ...

By the way, I guess Hobbes spot that ...

[bluemoon]

which means your Real Mode application would run in Protected or Long Mode and vice versa.

No, If you enabled long mode (setting MSR, either 64-bit long mode or 32-bit compatibility mode), you no longer able to natively run 16-bit code.
You can however implementa an emulator for that.

[0x57]

... recompile is almost unavoidable if you want to actually make use of the extra bits.

Thanks bluemoon. You're right, but I'm wondering how it could be possible to avoid that somehow, by just doing some "reasonable" stuff at/before run-time ...

[bluemoon]

You can avoid recompile of 16-bit application if you emulate the 16 bit environment - write your own or port DOSBOX; and provide/wrap/expose related APIs to it.

[0x57]

which means your Real Mode application would run in Protected or Long Mode and vice versa.

No, If you enabled long mode (setting MSR, either 64-bit long mode or 32-bit compatibility mode), you no longer able to natively run 16-bit code.
You can however implementa an emulator for that.

Well, I meant that you modify the binary before the actual execution to avoid that, so it would not use/need legacy stuff anymore and will looks like a normal guy (from the processor and OS prospective).

[bluemoon]

Well, I meant that you modify the binary before the actual execution to avoid that

Possible in theory, but binary translation is an very difficult topic of its own - its the core art of most emulator anyway.

[Brendan]

Hi,

If I make a Segmented memory in Real Mode, but later change it to a Paged one in Long Mode, I have couple of options like switching to the compatibility mode to run the old applications, right?

In long mode, you can execute code using the "16-bit compatibility submode"; and in protected mode you can use 16-bit code segments. This might allow you to run the same executable code (without recompiling) under both real mode and the "16-bit compatibility submode" of long mode and protected mode; as long as the code doesn't make any assumptions about the contents of segment registers (which also means that things like "malloc()" would have to be implemented in the kernel or in a shared library and can't be part of the application).

For example, typically real mode code will change segment registers (e.g. "offset++; if(offset == 0) { offset -= 16; segment++; }") to work around "64 KiB segment" limitations; and you'd have to make sure your 16-bit executables don't do that (in case the segment is actually a descriptor in the GDT). Of course this means that working with anything larger than 64 KiB (e.g. a decent size text file, a small array, a tiny photo, etc) will be a complete nightmare. Ironically, 16-bit code running in protected mode or long mode can use 32-bit data segments (where you wouldn't need to work around "64 KiB segment" limits if you didn't need real mode compatibility).

Mostly, in theory it's entirely possible. However, in practice you will never convince any sane programmer to write a 16-bit application and will also never convince any sane end user to use a real mode OS, so it's a complete waste of time to begin with.


Cheers,

Brendan

[0x57]

... its the core art of most emulator anyway.

I never thought about that! I would take a look at some resources to see how they're doing that like fail-safe process. I don't want to do some crap on every application right before launch it!

By the way, do you think a virtual machine approach would be a reasonable solution?

[0x57]

Mostly, in theory it's entirely possible. However, in practice you will never convince any sane programmer to write a 16-bit application and will also never convince any sane end user to use a real mode OS, so it's a complete waste of time to begin with.

Thanks Brendan, well just a little bit brainstorming here. That's always nice to measure your ideas publicly!

P.S. Theory and Practice should be the same, by the way ... however I know that they are not, anyways!