C standard library redesign

[bluemoon]

You are never out of virtual memory

In some system you may put a memory limit per process (or per user), and there is a case for out of memory.
Even on system without memory quota, you may still run out of virtual memory (ie. address space region dedicated to application) due to program bug.

[iansjack]

On the architectures that I program on, pointers are always a machine word in width so having an integral type, of whatever length, after a pointer will always be aligned. I'm assuming Jezze isn't spending alot of time on, to me, exotic architectures.

If the character part comes first then it will need padding to ensure that the length field is aligned. Pointers are a machine word width, but characters seldom are. If the length field comes first then you just have to ensure that the struct address is aligned.

I'm not seeing what you're getting at here. Can you give an example?

Knowing the address A of the struct the compiler knows that the length field is at A and the character field is at A + 4 (say). But if the character field comes first then it knows that the character field starts at A but has to compute the address of the length field. It is also marginally easier to manipulate strings (change length, concatenate, copy, etc.) if both fields are always at fixed positions relative to the struct address.

Small differences perhaps but, other things being equal, why not choose the most efficient representation.

But Owen's struct was:

struct { char* string; size_t length}

with the characters stored outside the struct.

Oops! You are correct.

[Combuster]

The main difference is that Owen posted a variant struct which contains a pointer to a string, rather than the actual string. In this case the string struct is of fixed size, but has the downsides of having the performance hit of an additional indirection. Common string types exists that actually stores both the length and string at the same memory allocation, in which case you have no option but to prefix the length if you don't want to end up calculating it as part of getting it back.

In the pointer case, putting length first might be a microoptimisation due to the logical read order (compute lengths, allocate, then copy strings if the pointer actually needs to be accessed), but there are similar arguments for the reverse order and no hard rules on the situation.

[Griwes]

Wait, so why do you even implement permission checks if it can't happen? Well, or do you think nobody needs new-fashioned stuff like permissions?

Right. I don't implement permissions of any sort, so it cannot happen.

Oh, another in the set of "rdos Fallacies": "I don't implement a check, so permission error will never, in any environment, on any system, written by any programmer, in any times, happen". I hope you realize how retarded that is; if not, you should be able to smell an ad hominem here.

Also, on most OSes I would expect that at least some data must be read before the program executes. And if it's only the header of the binary format. (While we're at it, it could also be a binary format that the OS doesn't understand. Another type of error.)

That's also non-recoverable. No error codes in the world could convert a non-executable file, or an executable file the loader doesn't understand, into something that suddenly can run.

Code: Select all

process foo("/foo/bar/baz.exe");
if (!foo.run())
{
     // ... how do I determine why it failed right in here? Oh, I don't, because one of yours non recoverable errors happened, and I can't even tell the user why. Too bad.
}

[XanClic]

When physical memory is exhausted, and no memory can be stolen from disk buffers, that's it. You won't get any further, and error codes won't help you.

You can do what Linux does: Kill non-vital applications.

Not too long ago I wrote some C macro code which caused the C preprocessor to eat gigabytes of memory. I was pretty glad Linux finally decided to kill it instead of rebooting the system or something similar.

[bwat]

When physical memory is exhausted, and no memory can be stolen from disk buffers, that's it. You won't get any further, and error codes won't help you.

You can do what Linux does: Kill non-vital applications.

Not too long ago I wrote some C macro code which caused the C preprocessor to eat gigabytes of memory. I was pretty glad Linux finally decided to kill it instead of rebooting the system or something similar.

How does the programmer/sysadmin communicate to Linux that a certain application is vital? For example, let us say I have a system where application A is vital, but application B is not. For the sake of argument assume B is dynamically allocating memory without freeing in an infinite loop. How can I make sure that in the case where resources become so scarce that progress cannot be made for B, that B and not A is killed?

[rdos]

Right. I don't implement permissions of any sort, so it cannot happen.

Oh, another in the set of "rdos Fallacies": "I don't implement a check, so permission error will never, in any environment, on any system, written by any programmer, in any times, happen". I hope you realize how retarded that is; if not, you should be able to smell an ad hominem here.

Other OSes that does have permissions need to solve the issue. I don't since I find the user/permission model lame and useless. You typically need keys and encryption to protect things, not permissions.

Code: Select all

process foo("/foo/bar/baz.exe");
if (!foo.run())
{
     // ... how do I determine why it failed right in here? Oh, I don't, because one of yours non recoverable errors happened, and I can't even tell the user why. Too bad.
}

Too bad? You mean you want a customer at a petrol station to see a pop-up dialog saying something like "Couldn't start foo because I had no permssion to use the file" or "Couldn't start foo becase the file is corrupt". What use would the customer have for this information? I'd expect to see something like "Terminal is closed".

[BMW]

Other OSes that does have permissions need to solve the issue. I don't since I find the user/permission model lame and useless. You typically need keys and encryption to protect things, not permissions.

We won't worry about trying to decrypt your encrypted data, we'll just delete it since there are no permissions.

[rdos]

Other OSes that does have permissions need to solve the issue. I don't since I find the user/permission model lame and useless. You typically need keys and encryption to protect things, not permissions.

We won't worry about trying to decrypt your encrypted data, we'll just delete it since there are no permissions.

That's fair enough. I could delete your permission based data simplying by booting RDOS on your target machine, and then deleting any file I'd like to delete. Alternatively, I could use the rmpart command to remove the complete partition(s) for you.

[bluemoon]

Permission and encryption are totally different things.

Since the introduction to ACL on grandfather's computer, now even OS like windows that usually used by single user provide two roles (user and administrator).
On Mac OS X (sandbox) or Android, you can also fine tune the permission of different aspect (access to internet, disk, user profile, etc)

Perhaps the world minus you think them useful.

[Jezze]

Sorry on so many levels. I misunderstood the structure, thought the char pointer was a buffer, I didnt explain the why and Im sorry for starting this discussion.

[Kevin]

Right. I don't implement permissions of any sort, so it cannot happen.

You're really making a brilliant point for a generic OS design discussion then...

When physical memory is exhausted, and no memory can be stolen from disk buffers, that's it. You won't get any further, and error codes won't help you.

If I as a user know that the problem was that I was out of memory, I can stop searching for the typo in the filename and instead close another program that's taking up memory. And after that I should very well get further.

That's also non-recoverable. No error codes in the world could convert a non-executable file, or an executable file the loader doesn't understand, into something that suddenly can run.

If it was a corrupted download, I can redownload the executable file. If it was for a different architecture, I can run it with my favourite emulator. But at least I don't have to look for the typo in the file name, check the permissions of the file (on many OSes, not including yours, of course) or close programs just to be sure that it wasn't one of the other errors.

Too bad? You mean you want a customer at a petrol station to see a pop-up dialog saying something like "Couldn't start foo because I had no permssion to use the file" or "Couldn't start foo becase the file is corrupt". What use would the customer have for this information? I'd expect to see something like "Terminal is closed".

I expect the log file to contain something more meaningful than "something went wrong".

[rdos]

Permission and encryption are totally different things.

Since the introduction to ACL on grandfather's computer, now even OS like windows that usually used by single user provide two roles (user and administrator).
On Mac OS X (sandbox) or Android, you can also fine tune the permission of different aspect (access to internet, disk, user profile, etc)

Perhaps the world minus you think them useful.

The point was that ACLs are part of software, so if I boot another OS that knows about the filesystem, but doesn't give a damn about ACLs, that other OS can do whatever it wants with the files. Or it could simply format the disk to get rid of the content. With encryption, you can very well delete the data, but without the key and algorithms, you cannot read protected data. Not so with ACLs.

[rdos]

Too bad? You mean you want a customer at a petrol station to see a pop-up dialog saying something like "Couldn't start foo because I had no permssion to use the file" or "Couldn't start foo becase the file is corrupt". What use would the customer have for this information? I'd expect to see something like "Terminal is closed".

I expect the log file to contain something more meaningful than "something went wrong".

Of course. First, the download manager will not even try to start the application if it's MD5 sum doesn't match the distributed checksum. That takes care of "file not found" and "corrupt file". And since it is started from boot, out of memory is not an issue, but if it is, the application starting it can query about free physical memory and other resources.

[XanClic]

When physical memory is exhausted, and no memory can be stolen from disk buffers, that's it. You won't get any further, and error codes won't help you.

You can do what Linux does: Kill non-vital applications.

Not too long ago I wrote some C macro code which caused the C preprocessor to eat gigabytes of memory. I was pretty glad Linux finally decided to kill it instead of rebooting the system or something similar.

How does the programmer/sysadmin communicate to Linux that a certain application is vital? For example, let us say I have a system where application A is vital, but application B is not. For the sake of argument assume B is dynamically allocating memory without freeing in an infinite loop. How can I make sure that in the case where resources become so scarce that progress cannot be made for B, that B and not A is killed?

If in doubt, you can always just kill the application which uses the most memory.
Most of the system programs should use relatively few memory, thus this should work pretty well. Alternatively, you could also have a special user (or profile or whatever) which is used to run the system applications (for Linux, this would probably be root) - applications which have been started by that user (or are running on that profile) simply aren't killed (or maybe last).