New segmentation technique in device-drivers

[rdos]

I'm right now moving a device-driver from assembler to C. This is the first time I move something I've written myself, and basically rewriting it in the process. The device-driver is the ini-file manager that apparently contains some bugs that I'm unable to locate.

The new design is starting to come up, and the structures are quite interesting as they are similar to how directories and files can be maintained in the filesystem driver.

Just doing a straight port gives a design like this:

1. Each ini-file is allocated it's own GDT selector, which results in appropriate limit checking and protection for the ini-file itself

2. Text from the file, sections and variable contents would need to be allocated from the small kernel memory pool, without selectors (they use flat addresses), basically breaking the whole idea of protection and isolation. The reason these cannot be allocated as selectors is that GDT selectors are a limited resource.

A slightly better approach is to place the text in the file itself at the end of the selector. The text would then be modified by inserting nulls at appropriate places, and using section names, variables and contents directly from this area. Then the section and variable structures would reference context in this area, and thus these pointers would also be protected. However, this still leaves the section and variable structures unprotected.

However, it is possible to place these are the end of the selector (after the text), and grow the selector as new sections and variables structures are added. A first implementation would probably set number of sections and variables to 0, and grow the selector when parsing the ini-file, as then the grow interface can be debugged and checked. After that, just searching for '[' and '=' would give a rough indicator of number of sections and variables, and this could be used to create a selector with the correct number of sections and variables before parsing, and so grow only needs to be done when adding sections or variables (uncommon).

This would create a fully protected design. An additional advantage might be that the "based" keyboard in OpenWatcom might be used, basically reducing the overhead of segment register loads to a single load upon entry.

[rdos]

You could probably utilize LDT to protect individual entries in the ini-file.

Cannot, since the ini-file interface needs to operate between processes. That's also why it needs to be in kernel space, not in application space. There is still the problem of changing content of ini-files with editors or by other operations in the filesystem. In the current implementation white space and other non-ini-file contents were preserved, but I won't do this in the new. When something is changed, the file will be written from scratch from the in-memory sections and variables. I'll probably synchronize from editors and alike by caching file modification time & size, and checking those from time to time.

[rdos]

just searching for '[' and '=' would give a rough indicator of number of sections and variables

Code: Select all

[entry]
value = '[[===[[[=======[=[[[[=[=[[[[[[==========[[[[[[===========[[[[[['

No problem. All it would mean is that the selector allocated would be larger than needed.

I've now finalized the parsing (using dynamic growth), and it works on a large Windows ini-file with 6 sections and 400 variables.

Now I need to code the "write-back" function that would recreate the ini-file from the in-memory copy. Then I should look at the modification code. I'm not sure about how to add sections and variables. One idea is to add them to the file text region, and just move the sections and variables. That could be implemented as part of the grow procedure.

The final result (when I've also modified the filesystem drivers) should be that no small kernel memory block should be accessed as "flat", rather all of them should be mapped to selectors and accessed in such a way that the allocated size would be enforced by hardware.

[rdos]

You could parse a file into an in-memory tree of key-value pairs, then manipulate it as you see fit from the applications and then reconstruct the ini-format from the tree representation on save. It makes more sense to me.

That's what I do, except that I don't want the code to be able to access anything outside of the region of allocated memory that I give it. Additionally, one large allocation uses a lot less memory than many small, and also is faster because the memory allocation needs to traverse block lists in order to get a free memory block. Freeing the object is also super-simple. All you need to do is to free the selector (which frees the memory block as well).

The grow algorithm doesn't add one item at a time (which would be bad for large files with dynamic growth), rather use the following algoritm:

Code: Select all

    Size = 3 * (Size + 1) / 2;

[rdos]

I've looked at the __based keyword in Open Watcom, and it offers possibilities to produce more efficient code, but there are a number of bugs in the implementation. The worst is that once the __based keyword is used in structures, the debugger no longer can display them correctly, and instead display content in default data segment. Another problem is that the compiler doesn't emit warnings when based variables are used without segment specification, making it a nightmare when these default to default data segment. OTOH, the code generated seems to be correct and quite efficient.

[bubach]

Send them some patches IIRC they can use all the help they can get. It's really a shame that they don't have more supporters, adding x86-64 support and keeping up with GCC. If I did more work in C, Watcom would probably have my support since I can't stand gcc & gas syntax.

[rdos]

Send them some patches IIRC they can use all the help they can get. It's really a shame that they don't have more supporters, adding x86-64 support and keeping up with GCC. If I did more work in C, Watcom would probably have my support since I can't stand gcc & gas syntax.

I can commit the patches myself, if I knew how to fix it. The problem is that I don't know the compiler well enough to provide the fixes. I posted on their news group for suggestions on how to fix it. Hopefully somebody knows a little more.

I got another idea that can at least "fix" the issues with the compiler automatically converting based pointers to far pointers relative to DGROUP. By using expand-down segments for based segments, these references would protection fault instead of silently overwriting DGROUP, which would be much better. That would also provide the needed guard-area around zero so null pointers can be approriately detected.