unite to make an OS

[nullplan]

That ties together the two topics of this post, because how would a DOS shell redirect program output to memory? By temporarily replacing the software interrupt vectors for output.

That was how DOS did a lot of things. For example, keyb.com is a program that adds support for different keyboard layouts (the default BIOS keyboard handler can only deal with American keyboards). It does that by overriding interrupts 15h and 16h, and also a couple of others. Or subst.com basically just overloading interrupt 21h, rewriting DOS system calls.

This is also DOS's biggest weakness: Once a TSR is added to the system, it cannot be removed again in a standard way. On Linux, you can kill a hanging process (unless it is hanging in non-interruptible sleep, but that is more of a hardware issue), and you can even unload kernel modules, but on DOS, a TSR is here to stay.

[eekee]

This is also DOS's biggest weakness: Once a TSR is added to the system, it cannot be removed again in a standard way. On Linux, you can kill a hanging process (unless it is hanging in non-interruptible sleep, but that is more of a hardware issue), and you can even unload kernel modules, but on DOS, a TSR is here to stay.

Good point. I (somehow) never had a hanging TSR, but had only good ones which had their own unload mechanism. Actually, that might not be true. It was so normal to reboot DOS computers that a TSR which didn't work well didn't make much of an impression. I would just try one and if it made the computer lock up, I would hit ctrl-alt-del and not try it again. As for the normality of rebooting, I've had to reboot FreeDOS many times because my text editor got funny. I had hopes of having fun with FreeDOS, even writing an OS with it, but a text editor which gets memory corruption if you move the mouse pointer while a key is down was one of the things which changed my mind.

[rdos]

Ew! I don't like that definition of autism either. It's more applicable to Asperger's syndrome because people with Asperger's syndrome can learn appropriate social behaviour, but Asperger's syndrome is no longer counted as part of the autistic spectrum.

I think a more accurate statement is that nowadays people that used to get the Asperger's syndrome (AS) label instead get an autism diagnosis. Which is a good thing since there never was a difference between the AS label and the autism label, and both are on the autism spectrum sharing similar issues.

Some people with some varieties of higher-functioning autism can can learn appropriate social behaviour too, but certainly not all.

Maybe some don't want to be appropriate?

[Gigasoft]

That ties together the two topics of this post, because how would a DOS shell redirect program output to memory? By temporarily replacing the software interrupt vectors for output.

The proper way to do this in DOS would be to use a temporary file residing on a RAM drive, instead of attempting to override a bunch of INT 21H functions.

[Alexey1994]

But if you dynamically calculate the relative address through the EIP register, then it will be possible to launch several processes, and one process will switch another through this same piping, that's when it becomes possible.

I don't understand this. Can you elaborate?

This is where I added the first implementation of what I had in mind https://github.com/Alexey1994/BelOS/com ... 836b286a31

This is not a fast implementation that switches the context of the process when reading or writing a single character. It also doesn't give a very good idea of the process, but this is only the first working implementation.

Code: Select all

cmd1 | cmd2 | cmd3

becomes:

Code: Select all

cmd1 >%TEMP%\cmd###1.tmp
cmd2 <%TEMP%\cmd###1.tmp >%TEMP%\cmd###2.tmp
cmd3 <%TEMP%\cmd###2.tmp

where % delimits environment variable names.

At first I wanted to do this, but I was motivated to do it right and now I can adapt programs to work with large amounts of data

[kerravon]

But if you dynamically calculate the relative address through the EIP register, then it will be possible to launch several processes, and one process will switch another through this same piping, that's when it becomes possible.

I don't understand this. Can you elaborate?

This is where I added the first implementation of what I had in mind https://github.com/Alexey1994/BelOS/com ... 836b286a31

This is not a fast implementation that switches the context of the process when reading or writing a single character. It also doesn't give a very good idea of the process, but this is only the first working implementation.

Thanks. I understand now, but note that I didn't see any mention of "EIP". Also note that "previous" doesn't have an "e" on the end.

Code: Select all

cmd1 | cmd2 | cmd3

becomes:

Code: Select all

cmd1 >%TEMP%\cmd###1.tmp
cmd2 <%TEMP%\cmd###1.tmp >%TEMP%\cmd###2.tmp
cmd3 <%TEMP%\cmd###2.tmp

where % delimits environment variable names.

At first I wanted to do this, but I was motivated to do it right and now I can adapt programs to work with large amounts of data

Would it be correct to say that your way is right (or perhaps ideal/good) for a single-tasking system? Actually, you've defacto created a scheduler, haven't you? Wouldn't it be better to formalize that rather than putting it in the piping process?

[Alexey1994]

Thanks. I understand now, but note that I didn't see any mention of "EIP".

Here https://github.com/Alexey1994/BelOS/blo ... main.c#L16
I calculate the offset relative to the EIP and further use it to calculate the absolute address of the data.

Actually, you've defacto created a scheduler, haven't you?

No, I don't have a schedule, just a context switch that launch only when needed (some event like alt+tab or lack of data in the pipe)

[kerravon]

No, I don't have a schedule, just a context switch that launch only when needed (some event like alt+tab or lack of data in the pipe)

Ok, so scheduling and context-switching are apparently different concepts.

But regardless, it probably doesn't belong in the piping mechanism.

I was thinking that when you have:

processA | processB | processC

that an efficient way of doing things would be for processA to do it's normal fwrite() which turns into a write() of BUFSIZ, although it may not detect that it is buffered, so it would be line buffered because it is stdout, and thus not be a full BUFSIZ. But we can start to address the situation where the 1-byte nature that you have implemented (as proof of concept I think) can start to be enhanced so that it would at least be line buffered. So it is at the point that write() or similar is called that you have multiple bytes, and the OS needs to determine what to do with them.

At this point, the OS has control, it hasn't wasted any effort doing buffering, and knows that this is a pipe scenario, and is in a position to do a context-switch to processB.

ProcessB would be initiated at this stage, and when it comes to do a read() of BUFSIZ, the data that was presented by ProcessA can be directly copied to satisfy its requirement.

The same "complication" of line buffering vs fully buffered would exist, plus the fact that the BUFSIZ may be different for the different processes (different compiler or compile options or even language).

And then presumably multi-processing/tasking/contexting/threading/whatever buzzword people keep using would sit on top of that. And a scheduler goes in somewhere.

[rdos]

My understanding of piping is that standard input is redirected to standard output of the previous stage. How the intermediate stages are implemented is up to the OS. In my design they are processes, but that is not necessary.

[iansjack]

That’s not strictly true. Pipes (in Unix at least) just provide a means of communication between processes via file descriptors. They are commonly used to replace the stdout descriptor of one process and the stdin of another, but that’s not the only way to use the connection. (For example, you might want to redirect stderr rather than stdout, or you might just want a connection totally separate from stdout and stdin.

[rdos]

That’s not strictly true. Pipes (in Unix at least) just provide a means of communication between processes via file descriptors. They are commonly used to replace the stdout descriptor of one process and the stdin of another, but that’s not the only way to use the connection. (For example, you might want to redirect stderr rather than stdout, or you might just want a connection totally separate from stdout and stdin.

Yes,that is my understanding too. I don't think pipes through file descriptors is an effective means of process communication. I have my own IPC mechanism that is far more efficient.

The main issue with file based communication is that files are stream-oriented, while communication typically is message oriented. There is no way to signal message boundaries with files, so various tricks are used. Particularly sockets are missing the "push" call that is a message mechanism, and typical socket implementations rely on timeouts instead of signalling that data should be sent.

[Alexey1994]

That’s not strictly true. Pipes (in Unix at least) just provide a means of communication between processes via file descriptors. They are commonly used to replace the stdout descriptor of one process and the stdin of another, but that’s not the only way to use the connection. (For example, you might want to redirect stderr rather than stdout, or you might just want a connection totally separate from stdout and stdin.

If you take the UNIX approach that almost everything is a file, then yes, stdin stdout is the only available means of communication.

But for me, a file is something that has a certain size and certain data as long as it does not change, so I do not think that it can be considered a file, for example, /dev/random, it is better to have a program that writes to some standard output, a pipe, arbitrary data.

I also believe that such a concept at the OS level as redirection of streams is not needed, since it is well done with the help of one program.

[kerravon]

There is no way to signal message boundaries with files, so various tricks are used. Particularly sockets are missing the "push" call that is a message mechanism, and typical socket implementations rely on timeouts instead of signalling that data should be sent.

Can fflush() be used? Which perhaps informs the OS via an fseek of 0 bytes from SEEK_CUR. Or alternatively, if you detect that you have switched from writing to a socket to reading from the socket, then that causes the "push"?

Not sure if it is relevant, but under PDOS I can do fopen("com1", "r+b"). And I do an fseek of SEEK_CUR 0 between reading and writing. C90 requires an intervening seek, of any kind.

[rdos]

There is no way to signal message boundaries with files, so various tricks are used. Particularly sockets are missing the "push" call that is a message mechanism, and typical socket implementations rely on timeouts instead of signalling that data should be sent.

Can fflush() be used? Which perhaps informs the OS via an fseek of 0 bytes from SEEK_CUR. Or alternatively, if you detect that you have switched from writing to a socket to reading from the socket, then that causes the "push"?

Not sure if it is relevant, but under PDOS I can do fopen("com1", "r+b"). And I do an fseek of SEEK_CUR 0 between reading and writing. C90 requires an intervening seek, of any kind.

I wrote my own socket API that includes the push function and that does not have an implicit timer.

I don't particurly like the idea that "everything is a file", so I created my own APIs and then (when useful) link them to file handles to achieve a bit of Posix compliance.

[eekee]

Some people with some varieties of higher-functioning autism can can learn appropriate social behaviour too, but certainly not all.

Maybe some don't want to be appropriate?

The main issue with file based communication is that files are stream-oriented, while communication typically is message oriented. There is no way to signal message boundaries with files, so various tricks are used. Particularly sockets are missing the "push" call that is a message mechanism, and typical socket implementations rely on timeouts instead of signalling that data should be sent.

In Plan 9, communication over pipes and virtual files preserves message boundaries; each write() makes a message. It's a great feature! It has a limit: Writes larger than 8KB will be split into multiple messages. (The OS uses 8KB buffers in many places. )