Guys what is the purpose of handles ?

[klange]

I think the only commonality among the myriad uses of the term "handle" is the idea of opaqueness: You're not supposed to attempt to interrogate a handle, only give it to the interfaces its "owner" provides you. The concept of abstraction at the level of different handles doing different things for those interfaces is secondary to that opaqueness.

[Solar]

Thank you.

[rdos]

I think the only commonality among the myriad uses of the term "handle" is the idea of opaqueness: You're not supposed to attempt to interrogate a handle, only give it to the interfaces its "owner" provides you. The concept of abstraction at the level of different handles doing different things for those interfaces is secondary to that opaqueness.

Using that definition, I feel Posix file handles doesn't qualify.

1. You can interrogate them (although, through cludges)
2. The owner doesn't provide you with an interface
3. Not all functions in the "interface" can operate on all handles (for example, positioning doesn't work on stdin or stout)

Additionally, the Posix handle tables & numbers are maintained by the user library, and due to the presence of dup, the library might have its own table that translates between handles and "OS handles" (if the latter exists at all). It's actually very unclear (and OS dependent) how this stuff is implemented in the C library.

[Solar]

So the example isn't perfect. (Just like your understanding of what "opacity" actually means.) But it is intuitive and good enough to give an idea about handles, especially since we can assume everybody having encountered both POSIX file I/O and standard C file I/O previously. Which is more than could safely be assumed for USB device drivers.

Now.

Could we perhaps return to devc1's actual problem, or do you feel this thread needs yet more derailment on the relative merits of example A over example B?

[klange]

Using that definition, I feel Posix file handles doesn't qualify.

What is a "Posix file handle"? I don't know if you mean a file descriptor or a FILE*. The latter is not a POSIX invention, it is standard C. They are representative of two very different approaches to handles, but they are still both handles.

1. You can interrogate them (although, through cludges)

You can not interrogate a file descriptor - to the application it is just a number. You can only affect the resource it refers to using the interfaces provided by the system.
You are not intended to interrogate a FILE* - the structure it points to does not have an exposed definition. That it is just a pointer to a struct in memory does not make it not a handle.

2. The owner doesn't provide you with an interface

For a file descriptor, the "system" is the owner, and the interface it provides is the system calls that take file descriptors as arguments (or return them, or use them in other structures).
For a FILE*, the owner is the libc. The interfaces are all of the libc functions that deal with those FILE*s. That this lives in the same memory space as the user of the handles is irrelevant.

3. Not all functions in the "interface" can operate on all handles (for example, positioning doesn't work on stdin or stout)

That's irrelevant to whether either of these are handles. Handles do not imply any universality of interfaces.

Additionally, the Posix handle tables & numbers are maintained by the user library, [...]

File descriptor tables are not at all maintained by the "user library", nor are the numbers.

[...] and due to the presence of dup[...]

dup is purely part of the file descriptor API and does not interact with the FILE API. The libc is not involved in what happens behind the scenes to the file descriptor table when dup is called. As an example, with glibc on Linux, dup only calls the fcntl system call (through a wrapper that sets up some syscall cancellation context that isn't important), examines the result for an error code to set errno, and otherwise returns the resulting file descriptor. It does not track it in anyway, nor does it have influence over what file descriptor will be returned.

the library might have its own table that translates between handles and "OS handles" (if the latter exists at all).

On a POSIX system, FILE objects will most definitely have a file descriptor in them somewhere (if they reference a file that works that way - fmemopen is a whole different ordeal), and the FILE APIs will most assuredly involve calling the relevant POSIX APIs using that file descriptor, but that is a one-directional relationship. Generally, FILEs may be stored or tracked in some sort of list (in order to support automatically flushing at exit), but this list does not represent a table by which a file descriptor may be mapped to a FILE. Further, if you use fileno to obtain the file descriptor backing a FILE, the FILE APIs are oblivious to things you do "behind its back" with that file descriptor and intermixing the two APIs with the same resource is unspecified.

It's actually very unclear (and OS dependent) how this stuff is implemented in the C library.

That is the entire point: What the owner of the handle is doing with it to implement the functionality it is exposing to you is none of your business as the user of the handle.

[rdos]

You can not interrogate a file descriptor - to the application it is just a number. You can only affect the resource it refers to using the interfaces provided by the system.

Yes, you can. You can interrogate file descriptors with ioctl.
Second, since the file descriptor is a feature of the usermode libc, you potentially can affect it by corrupting memory. The same applies to FILE *, where you can also overwrite the pointer data. Real OS-level handles cannot be corrupted like this since they are handled exclusively in kernel.

For a file descriptor, the "system" is the owner, and the interface it provides is the system calls that take file descriptors as arguments (or return them, or use them in other structures).

Wrong. It's libc that is the owner, and the C runtime interface is not based on system calls. At least, doesn't need to be.

In my initial implementation of file handles, I put an array in libc that linked C file handles to either stdin, stdout or a real file handle. IOW, file handles where completely implemented in libc, and based on file type, would do syscalls either to keyboard, console or a file. Thus, it's perfectly possible to implement C file handles without passing the C file handle to kernel.

For a FILE*, the owner is the libc. The interfaces are all of the libc functions that deal with those FILE*s. That this lives in the same memory space as the user of the handles is irrelevant.

It's relevant from the perspective of protection from accidental or malicious manipulation.

File descriptor tables are not at all maintained by the "user library", nor are the numbers.

That's implementation specific. They can be maintained by libc, they could just redirect to the OS kernel, or anything in between.

[rdos]

Could we perhaps return to devc1's actual problem

That would be a good idea.

So, the purpose of handles is to hide implementation details in kernel from users of a handle. Another purpose is to create well-defined interfaces so the user know what he/she can do with the handle. In OO, this is called encapsulation and classes.

You know you have done a good job with the interface if you can create a clean C++ class that hides the handle and implements all the methods in the interface where the handle can be used.

You know you have done a poor job if you end up with a mess of linked C++ classes, enums, struct definitions and lots of void * pointers, undefined function codes and alike. You also know the design is poor if some handles result in undefined behavior with some methods.

Here is an example of how to implement a file class:

Code: Select all

class TFile
{
public:
    TFile(const char *FileName);
    TFile(const char *FileName, int Attrib);
    TFile(const TFile &file);
    ~TFile();

    int IsOpen();
    const char *GetFileName();

    long long GetSize();
    void SetSize(long long Size);
    long long GetPos();
    void SetPos(long long Pos);
    TDateTime GetTime();
    void SetTime(const TDateTime &time);

    int Read(void *Buf, int Size);
    int Write(const void *Buf, int Size);
    int Write(const char *str);

protected:
    void VfsLock();
    void VfsUnlock();
    int VfsReadOne(int index, char *Buf, long long Pos, int Size);
    int VfsFind(long long Pos);
    int VfsRead(void *Buf, int Size);

private:
    long FHandle;
    char *FFileName;
    struct FileMap *FMap;
    int FMapIndex;
    int FLastIndex;
};

This class both works with my new memory mapping scheme, and by passing the file handle directly to the appropriate syscall.

[klange]

Yes, you can. You can interrogate file descriptors with ioctl.

ioctl is an interface provided by the owner of the file descriptors, so using it is very specifically not what I mean by "interrogate".

Second, since the file descriptor is a feature of the usermode libc, you potentially can affect it by corrupting memory. The same applies to FILE *, where you can also overwrite the pointer data. Real OS-level handles cannot be corrupted like this since they are handled exclusively in kernel.
[...]
Wrong. It's libc that is the owner, and the C runtime interface is not based on system calls. At least, doesn't need to be.

As much as POSIX does not delineate these things, in the original Unix, in BSDs, in macOS, in Solaris, in Linux... file descriptors are entirely owned by the kernel, and not at all by the userspace libc, and the interfaces for dealing with them are all system calls.

In my initial implementation of file handles, I put an array in libc that linked C file handles to either stdin, stdout or a real file handle. IOW, file handles where completely implemented in libc, and based on file type, would do syscalls either to keyboard, console or a file. Thus, it's perfectly possible to implement C file handles without passing the C file handle to kernel.

Okay. They're still handles.

It's relevant from the perspective of protection from accidental or malicious manipulation.

Which is irrelevant to them being handles.

That's implementation specific. They can be maintained by libc, they could just redirect to the OS kernel, or anything in between.

And, again, that is still irrelevant to them being handles.

Handles that strongly enforce opaqueness, that are completely immune to interrogation, that can not be spoofed or forged, are better known as capabilities.
Handles that are little more than pointers to things you aren't supposed to have the definition for are called opaque pointers.
Both are still handles.

[rdos]

ioctl is an interface provided by the owner of the file descriptors, so using it is very specifically not what I mean by "interrogate".

Ioctl is an undocumented kludge that can do anything to the file descriptor, including providing hints so the handle can be interrogated.

Handles that strongly enforce opaqueness, that are completely immune to interrogation, that can not be spoofed or forged, are better known as capabilities.

OK. I prefer to implement capabilities, that are based on handles.

[klange]

Ioctl is an undocumented kludge that can do anything to the file descriptor, including providing hints so the handle can be interrogated.

A horrible, garbage interface is still an interface.

Also, if you re-read my original post, I said that you aren't supposed to interrogate a handle, not that you necessarily can't. Intent is a key thing here.

"Interrogate" here is allegorical. If you throw the handle in a locked room, with no access to its interfaces, can you get useful information out of it? A capability will never squeal because it can't, it knows nothing. An opaque pointer may spill a coded secret, but it doesn't know what it means, and you'll only know if you stole the codebook. A non-handle - a transparent pointer to a defined struct - will tell you everything and not even ask for a plea deal.

OK. I prefer to implement capabilities, that are based on handles.

For these sorts of things, I don't disagree, but, responding to something from your previous post...

the purpose of handles is to hide implementation details in kernel from users of a handle

I think you're fixating on a particular use case. Handles are not exclusive to kernel-user separation.

Handles do very commonly refer to OS resources, and strict enforcement of separation is a boon there. Capabilities are great!

But "handle" is an extremely generic term, which is why I phrased my original post the way I did: the only thing various kinds of handles have in common is that you, as the user of the handles, aren't supposed to care about their contents because that is the exclusive domain of the thing that owns the handle. That is what opaque means.

[devc1]

Guys can you please stop arguing and tell wether my handle manager or object manager is good to go.

[iansjack]

Surely it’s up to you to judge whether your API does what you want it do.

[linguofreak]

Guys can you please stop arguing and tell wether my handle manager or object manager is good to go.

The issue here is that "handle" is just a word. It has been used to refer to different somewhat-related concepts in different systems. The people arguing are familiar with the definitions used by particular systems and are arguing about which one the word 'handle' "really" refers to.

But the word "handle" itself is not magic. Given the title of this thread, it sounds like you don't have a firm grasp on any of the concepts that the word "handle" has been used to refer to, and you have written a handle manager because you heard somewhere that you need a handle manager without understanding why. To give you any real help, people will need to know the following things:

1) What is a problem that you are running into in the design of your OS that you think might be solved by "handles"?
2) Forgetting about the name "handle", or any other name we might give to it, what is a potential object/data structure/whatever that you could use to solve this problem?
2a) Are there other problems that this concept could solve for you?
2b) To the best you can tell, does your handle manager actually implement the concept described in 2) ?

If the answer to 2a) is "yes", then your handle manager is likely to implement a useful, generalizable concept that you can use to solve many problems. If the answer is "no", then you should consider if you can find other concepts that solve 1) that can also be used to solve other problems. Otherwise, in the long run, you risk having multiple, incompatible handle types for different uses that need different handle managers.

If the answer to 2b) is "no", then you yourself have established that your handle manager is not good to go.

If you have answers to 1) and 2), and if you can answer 2b) with "yes", then people can tell you if your handle manager is any good. They can tell you if your answer to 2) really solves problem 1), and they can tell you if the answer to 2b) really is yes, or if you've made a mistake.

If you don't know the answer to any of these questions, then people in this thread can only really do two things:

1) Have the argument that has already been had about what a handle is.
2) Tell you that different systems have called all of the concepts discussed in the argument "handles" (and a few systems have probably used "handle" to describe things not discussed in the argument) at various times in the past, and that "handle" isn't a magic word, just a word.