Interchangeable device drivers

[turdus]

Quite a time ago I run into UDI. Unfortunately is seems to be a dead project mostly for it's complexity.

But I really like the idea of sharing device drivers among hobby kernels, so I came up with Universal Driver Block Format, which is
* kernel design, executable format and tool-chain independent
* gives binary compatibility for hobby kernel drivers
* simple as it can be, yet flexible
* easy to implement in C and assembly
* easy to add to your existing drivers
* provides a way to update your drivers

What's your opinion, does it worth keep working on?

[thepowersgang]

(Haven't yet read your attempt at a spec, I'll take a look this evening and post again then)

UDI is not dead, I've attempted to lead a slight revival, and the #udi channel on Freenode has some active people. I'd suggest porting (or if you're very brave, writing your own) a UDI environment and giving that a fly instead of attempting to design another "interchangable" device driver format.

[Owen]

Also, your design looks like a crippled version of CDI.

Those of us who lean towards UDI lean towards it because it is good. This is not a slight against CDI; CDI has the aim of being simple, but that has the contrasting implication that the price for simplicity is poorer performance characteristics. The performance of UDI is up there with some of the best designed native driver interfaces.

[gravaera]

Yo:

As tPG said, you can /join #udi on Freenode, and we have around 2-3 people in there at all times who understand the specification very well, and can give you very hands-on advice on how to go about adding support for UDI in your own kernel, regardless of your kernel's design. At least 2 of the people who are constantly in the channel are implementing UDI environments themselves within very mature kernels.

--Peace out,
gravaera

[Kevin]

Quite a time ago I run into UDI. Unfortunately is seems to be a dead project mostly for it's complexity.

But I really like the idea of sharing device drivers among hobby kernels, so I came up with Universal Driver Block Format

Not counting yourself in either project, how do you think UDRV is more alive than UDI or CDI? And of course this one must not be missing: http://xkcd.com/927/

Not sure where exactly you want to position this spec, it seems to be mostly CDI, except written down worse (don't use uint32_t for function pointers...). It tries to take CDI's simplicity and "good enough" approach and add some of UDI's points like binary compatibility (perhaps that's actually the only one).

Unfortunately, you didn't really consider that binary compatibility is hard: A major problem that I see with your proposal is extensibility. UDI solves the problem with adding quite a bit of complexity (well, at least I hope that the whole complexity can solve this, I'm not familiar enough with the details). CDI "solves" this problem mostly by not caring about binaries at all. In your case, UDRV has a given set of function pointers set in stone (you don't even have an interface version number that would allow compatible extensions at least) and this will have to be enough forever. Looking at the set of functions you have, however, I immediately see that it's far from complete.

Unless you can name a few real advantages of UDRV over both CDI and UDI, my advice is switching to one of the existing projects. Activity isn't one, UDRV doesn't even exist without you.

[thepowersgang]

After taking a look (and seeing the other comments), I'm inclined to agree that this is not the best solution [ugh, uint32 as pointers? At least you could use the standard datatype]

[turdus]

Thank you for the answers.

@thepowersgang: are you planning to use executables over 4G?

[Combuster]

How about 0xfffffffffffc0000?

Native pointer types save you from explicitly having to do the truncating and zero/sign-extending. Plus casting from function pointers to non-function types is forbidden in C.

On to a completely unrelated section, there is no proper definition of "fastcall" for x86_64.


Basically, the obvious sanity checks fail which would leave everyone attentive enough to wonder what other less obvious design errors there are.

[Owen]

"There's no restriction on it's format (dll, so, kext etc. use whatever you like, see shared library). "

Hang on, so this is CDI, but worse, with no intent at binary compatibility, yet with a crazy boneheaded header format (Which can't even represent pointers on 64-bit architectures, mind you!), why?

If you want simple, use CDI. If you want highly performant and flexible, pick UDI.

This is neither, has a wholly underspecified interface, and no drivers. Why?

[turdus]

How about 0xfffffffffffc0000?

Native pointer types save you from explicitly having to do the truncating and zero/sign-extending. Plus casting from function pointers to non-function types is forbidden in C.

What are you talking about? Who told you about casting anything? It seems you missed to read that part that states that the linker's duty to convert 32 bit file offset into a 64 address after the load address is known. The point is to keep the file overhead as small as possible (drivers bigger than 64k? maybe, over 4G? very unlikely), and it won't affect your performance because you extend only once on load, and use the 64 bit entry table afterwards (each with a predefined prototype, no casting involved at all).

On to a completely unrelated section, there is no proper definition of "fastcall" for x86_64.

By fastcall I mean passing arguments in registers, see here http://www.x86-64.org/documentation/abi.pdf, section 3.2 Function Calling Sequence.

Basically, the obvious sanity checks fail which would leave everyone attentive enough to wonder what other less obvious design errors there are.

Basically you are criticizing without proper reading.

Moderators: as the community does not like the idea of OS and toolchain-independent information and entry-point block in drivers, I'm not going to work on this, you can close this thread. Thanks.

[turdus]

Hang on, so this is CDI, but worse, with no intent at binary compatibility

No, implements only binary compatibility.

yet with a crazy boneheaded header format

That can be constructed with every language, every compiler and every executable format, which was the point. UDRV does not have source level definitions, focuses merely on an executable which is called device driver.

(Which can't even represent pointers on 64-bit architectures, mind you!), why?

Because they don't have to. They are offsets within the binary, and not memory addresses. They converted to memory addresses when your kernel loads the driver, and once the load address known.

If you want simple, use CDI. If you want highly performant and flexible, pick UDI.

CDI does not cover half of what UDRV was designed to do.
1. You can use a CDI driver only and only if your kernel can load it's binary format.
2. It categories drivers in a static way, while UDRV uses a well-known and open category for that. For example you cannot write a scanner driver for CDI as of now (take a look at include/cdi.h, no such category).

UDI:
1. wikipedia states that it's a "defunct project". Describes with only past sentences.
2. http://projectudi.sourceforge.net/ not changed in 10 years
3. it's way too complex and overcomplicated for a hobby kernel
4. if you want to utilize even the simpliest UDI driver with your kernel, well, that's hell a lot of work. No newcomers can benefit from it.
5. try to find drivers: google "open source udi driver" does not give you even a single downloadable driver on the first 3 page (not checked the 4th). Only papers, opinions and talks about it, most of which couple years old.
6. was created by manufacturers, not hobby kernel developers

Neither CDI, nor UDI can:
1. detect if a driver applicable to your system without executing the driver
2. has standardized command line interface to configure the devices
3. driver update distribution solution, because (let's face it) we are a developer community with very few "finished" code.

This is neither, has a wholly underspecified interface, and no drivers. Why?

Underspecified? No. Not put all in the wiki before asking your opinion? Yes...
Well, I've made my points, hope answer some of your questions.

Nevertheless seeing the OS community resistance makes me to abandon it.

[Kevin]

That can be constructed with every language, every compiler and every executable format, which was the point. UDRV does not have source level definitions, focuses merely on an executable which is called device driver.

In fact, the source level things are what I was still waiting for. You started to outline that some library exists, but there are only very few definitions (uncommented, too).

If I were to implement a UDRV file system driver, I would want to access the disk from the driver. Or if I were to write a disk driver, I would want to find the PCI device, enable PCI bus mastering and then access I/O ports and MMIO. How would I do this? Your wiki page doesn't say anything about this.

1. You can use a CDI driver only and only if your kernel can load it's binary format.

CDI is source based. You simply compile the driver into a binary that is useful for your OS.

2. It categories drivers in a static way, while UDRV uses a well-known and open category for that. For example you cannot write a scanner driver for CDI as of now (take a look at include/cdi.h, no such category).

Then add one. As long as there is no scanner driver, there is no reason to have a #define for the category. The first patch series that adds a scanner driver will also make the necessary extensions to the header files.

(By the way, UDRV doesn't really support scanners either, or which branch of the union would it take in udrv_header_t?)

[Owen]

Hang on, so this is CDI, but worse, with no intent at binary compatibility

No, implements only binary compatibility.

I reiterate: "There's no restriction on it's format (dll, so, kext etc. use whatever you like, see shared library). "

(Which can't even represent pointers on 64-bit architectures, mind you!), why?

Because they don't have to. They are offsets within the binary, and not memory addresses. They converted to memory addresses when your kernel loads the driver, and once the load address known.

While the binary is unlikely to occupy 4GB, I regularly run binaries where there is a greater than 4GB difference between the load address of the lowest segment and the end of the binary

If you want simple, use CDI. If you want highly performant and flexible, pick UDI.

CDI does not cover half of what UDRV was designed to do.
1. You can use a CDI driver only and only if your kernel can load it's binary format.
2. It categories drivers in a static way, while UDRV uses a well-known and open category for that. For example you cannot write a scanner driver for CDI as of now (take a look at include/cdi.h, no such category).

So add a category?

UDI:
1. wikipedia states that it's a "defunct project". Describes with only past sentences.
2. http://projectudi.sourceforge.net/ not changed in 10 years

http://projectudi.org

You linked to the reference implementation (For Linux, FreeBSD, etc), not the project. You'll note that website was revised in 2010.

The spec doesn't really need active revision. There are some errata, but everything that needs doing (Metalanguage definition, for example) can be done without needing to wake up the slumbering giant.

3. it's way too complex and overcomplicated for a hobby kernel

Matter of opinion. "UDRV" is far too simplistic for anything but the most basic of hobby kernels (It appears to use synchronous interfaces, ye gods!)

4. if you want to utilize even the simpliest UDI driver with your kernel, well, that's hell a lot of work. No newcomers can benefit from it.

Supporting newcomers at the cost of being useless to everyone else was never the goal

5. try to find drivers: google "open source udi driver" does not give you even a single downloadable driver on the first 3 page (not checked the 4th). Only papers, opinions and talks about it, most of which couple years old.

I don't see any UDRV drivers yet either.

The list of UDI drivers isn't great, though most of us who look to UDI consider any drivers that arise more of a bonus than the reason we go there (We go there because smart people already went there and did a lot of the hard thinking for us - no reason to reinvent the wheel badly)

That said, for members from our community, both thePowersGang and Combuster have some UDI drivers in various states embedded inside their repositories.

There are also a few UDI drivers lying around for use with UnixWare. I'm not certain of the licensing of them (Except that they're closed source and probably i386 binaries).

6. was created by manufacturers, not hobby kernel developers

It was created by people who understood what they were doing. It was created with the intent of being the driver interface to end all driver interfaces.

It is one of the best authored technical specs I have ever read, if occasionally slightly hard to understand

Neither CDI, nor UDI can:
1. detect if a driver applicable to your system without executing the driver

Lack of cursory research on UDI: Check

2. has standardized command line interface to configure the devices

The commonality between, say, the UNIX and VMS command line interfaces is very slim.

Unless your intention is to support POSIX and POSIX systems alone, not very useful.

Besides: Why am I configuring the driver? That tells me something has gone wrong somewhere.

(And why am I not using a GUI, anyway?)

3. driver update distribution solution, because (let's face it) we are a developer community with very few "finished" code.

UDI has a standard package format.

For CDI... git pull

Underspecified? No. Not put all in the wiki before asking your opinion? Yes...

The "specification" has lots of holes and lots of wavy language.

Its' incongruous to claim that it has features that CDI doesn't have when you haven't even written out your own specification of those features

Nevertheless seeing the OS community resistance makes me to abandon it.

There are already two standards: CDI, if you want something easy to implement (All credit to CDI: As a technical exercise it may lack, but that's because it has different aims). If you want something which can scale (For example, you consider your project 'serious' for whatever reason: I for example consider my own a 'research project', which would be hampered by an insufficiently capable driver model) then the standard of choice is UDI.

The only concrete advantage you've offered is binary compatibility. Which is fine, but I don't see why you couldn't add that feature to CDI. After all, there has to be some "driver description" structure somewhere. I'm sure you can find it and figure out how to export it from an object file.

[Kevin]

I agree with pretty much everything you said, Owen, except for the part with adding binary compatibility to CDI. You can, of course, do that for a single specific CDI revision, but when talking about binary compatibility, you usually mean across versions. And that imposes a lot of restrictions on the kind of changes that can be made for a benefit that the existing CDI users don't care about.

That said, UDRV doesn't have any mechanism that would allow compatibility across versions either, so I guess it doesn't matter too much.

[XanClic]

I believe you can't even have binary compatibility for CDI in just a single version across different operating systems. Basically, CDI only defines what fields a structure must contain, but there is no limit, so you're in practice always free to add own OS-dependent members to CDI structs, if need be (in theory, though, one shouldn't just wildly add own fields to all the structures). This is standardized through the *osdep structs and for different OS, these may differ and therefore binary compatibility can never be guaranteed.

But this was just a design decision for CDI in the first place: We'll probably have no need for binary-only drivers, so we can exploit this for greater flexibility. I don't see the problem: If you want to have binary compatibility, you give up on this flexibility. Not having binary compatibility can therefore be seen as a feature as well.