Designing a hard-realtime addon to an operating system

[rdos]

I have a very specific application in mind for this. Imagine that you have a 1G sample AD converter card (they typically come as PCI-cards that should be plugged into graphics grade PCI slots), and want to analyze up to a minute or so of data for unknown patterns. First, I will just want to collect 100Gbytes or so of realtime data and do the analysis in retrospect. However, as I get to know the data, I probably would want to do at least part of the analysis in realtime and reduce the data to some other form that is much smaller. Typical designs would add the logic to the data acquisition equipment since typical operating systems like Windows and Linux simply cannot handle this in realtime.

I looked for hardware suitable for this project, and a promising setup is a two CPU AMD EPYC board. I could populate it with two rather low-end EPYC processors to begin with, and then upgrade them if necessary. This board also has dedicated RAM for each processor, and so the amount of installed RAM could be different between the analyzer and the operating system.

The realtime analyzer should work with physical memory and not linear memory (unity mapping). This is largely incompatible with modern operating systems but should be possible to implement if the realtime system run on dedicated core(s) in its own mode. Given that the function must be able to access huge amounts of RAM, it will need to run in long mode regardless of the mode used in the operating system.

I envision that there needs to be a communication interface with the operating system. The realtime system can send interrupts to an operating system driver. For one, the realtime system must allocate physical memory, preferently in large blocks, like 100GB of continous RAM. It must also be possible to debug the realtime system from the operating system driver. When the realtime system encounter faults it should stop and inform the operating system driver. It should be possible to singlestep the realtime system code, which can be done with the trap interrupt. It must also be possible for the operating system driver to put the realtime system into debug state, for instance by sending it an interrupt.

I don't want to do this in Linux or Windows, rather in my own operating system. I have a simulator, so the first step probably is to run the realtime system code in a simulator, tying read/write of physical memory to real physical memory. Typical simulators will not work since I need to debug the interaction between the operating system and the realtime system. Once I have the debug interface done, I can then use that for debugging the realtime system instead of using a simulator.

[Solar]

Is there a question in there somewhere? Because if it is, I can't find it.

Generally speaking, a system is either hard-realtime or it isn't, and if it isn't, you can't bolt it on retroactively. When your HRT part is sending interrupts to a non-HRT part, your HRT part isn't HRT anymore because the non-HRT part can't guarantee that the interrupt will be handled in the available time slice. And that kind of guarantee is what makes hard realtime "hard".

[rdos]

Is there a question in there somewhere? Because if it is, I can't find it.

It was mostly ideas on how to realize a specific research project given that I cannot find a way to do it with traditional systems.

Generally speaking, a system is either hard-realtime or it isn't, and if it isn't, you can't bolt it on retroactively. When your HRT part is sending interrupts to a non-HRT part, your HRT part isn't HRT anymore because the non-HRT part can't guarantee that the interrupt will be handled in the available time slice. And that kind of guarantee is what makes hard realtime "hard".

Not necessarily. Many modern hardware devices can operate in realtime and still send interrupts to the target operating system. A good example is a USB controller, a network card or an AHCI disc controller. I think ideas from that environment can be used for how to exchange data without breaking realtime performance. For instance, devices like that typically work with memory-based schedules, and interrupts are only used for signaling changes to the schedules themselves which the operating system will process in parallel. The operating system also does the configuration and starts or stops the realtime device. The difference here is that the realtime device is not a dedicated hardware device, rather is run on one or more cores in the computer. Something that is much more flexible than dedicated hardware.

[LtG]

What's the problem with using Win/Lin? The description doesn't sound like HRT is needed.

Either it's 100GB/s or 100GB/min, the latter is only 1.5GB/s, which shouldn't be a problem with Win/Lin.

[LtG]

When your HRT part is sending interrupts to a non-HRT part, your HRT part isn't HRT anymore because the non-HRT part can't guarantee that the interrupt will be handled in the available time slice.

Whether the OS handles the interrupt within HRT constraints is irrelevant if the HRT part itself isn't dependent on the interrupts result. For instance the HRT part using interrupts to signal to the OS to update GUI.

Also, is there some significant issue "bolting on HRT"? For instance dedicating some cores to HRT shouldn't be that difficult.

[rdos]

What's the problem with using Win/Lin? The description doesn't sound like HRT is needed.

Either it's 100GB/s or 100GB/min, the latter is only 1.5GB/s, which shouldn't be a problem with Win/Lin.

A sample is 16 bits, and with a sample frequency of 1GHz, it means just reading the data would be 2GByte/s. I think the AD conversion card can buffer data in RAM, but realtime processing that should be sustained for a minute or so cannot tolerate that the OS runs other things at the same time.

[Schol-R-LEA]

Not necessarily. Many modern hardware devices can operate in realtime and still send interrupts to the target operating system. A good example is a USB controller, a network card or an AHCI disc controller.

None of those are operations which usually have real-time constraints, not even soft ones. Online constraints, yes, but not real-time ones in a technical sense.

Generally speaking, RT is only going to apply to something where there is an output control signal being sent, rather than an input data signal being received. If the input is necessary for the system to decide how to proceed with a later RT operation, then yes, processing it may have a real time constraint, but even then that isn't necessarily the case. Even if it is, from the POV of the operating system, the constraint usually applies to the process (driver, application, etc.) which is creating the control output, rather than the input itself.

Also, a control output is only real-time if the operation must be performed within a fixed time window of no earlier than t and no later than t' to avoid unrecoverable failure of the operation itself - the lower time boundary is just as important as the upper one when discussing RT (even if it is defined as '0 nanoseconds', it is still a lower boundary), as is the requirement that a failure leads to some irrevocable outcome (which could be anything from a paper jam to a power plant meltdown, so long as it is something which cannot be ignored or undone).

Of those examples, only the USB controller is likely to have an RT requirement, when operating peripherals such as mice where timing is a factor, be even there the usual goal is 'fast as possible response' rather than 'response within a fixed time window' - the online timing is handled in software after the signal is received, rather than as part of the signal capture, so the operation itself is not, strictly speaking, an RT one, or even an online one.

Also, a slow response for such a device (or slow processing of the input signal by the driver or application, or even an outright failure to receive the signal) usually won't cause an outright failure - e.g., it may cause you to miss a shot in CS:GO or Fortnite (assuming you're a video gamer, I have no idea if you are or not), but honestly there are many other causes within games like that themselves which are far more likely to have that effect, and more to the point, such a stutter won't usually cause the game to abort or crash. The game as a whole will still proceed, even if you lose the match because of that stutter.

More important for our discussion, though, is the fact that the time-critical aspects of the actual peripherals (if they have any) are mostly handled in the hardware or firmware of the devices themselves, rather than as part of the driver in the OS. A number of things which historically were considered soft real-time because they were handled in software on the parent system (e.g., the timing of a printer's write head, or of a floppy drive's read/write head on systems such as the Apple II) were long since moved into dedicated hardware, and even before that, many of them didn't really have a hard real-time constraint - for example, if a disk's R/W head overshot, the driver would usually be able to detect this and adjust it before committing to a read or write operation.

This means that your later example of AD sampling really isn't RT either, unless you are doing the conversion online without any sort of buffering. Again, that is likely to be part of the sampling hardware. Even if you are sampling online ('in real time' in the colloquial sense), it is unlikely that it has a real-time constraint in the software development sense - I'm not sure that a stutter would be considered an irrevocable failure, unless you are performing live for an audience, or if it causes a recording of a unique event to be unusable.

And that's sort of the crux of this issue: most things which are online are not, in fact, real time in the technical sense. Yes, things such as video compositing should be done within a time constraint, but the operation won't fail irrevocably if it isn't. It won't cause a line of text to be misprinted in a software-driven printer (which, as I said earlier, was a classic example of a 'soft real-time' constraint in the 1970s and 1980s - a guarantee of 'best effort' performance was acceptable even if it caused paper and ink to be wasted), cause a paper jam (a slightly less soft RT constraint for the same sort of system), knock a bottle over and force an assembly line to halt ('firm' real-time - it is mission critical, and the timing restraint is a hard one, but it is unlikely to be life-threatening unless someone seriously messes up elsewhere), or cause an aileron to get stuck resulting in a plane crash (a genuine hard real-time constraint, where failure leads to significant real world consequences).

In other words, it doesn't sound as if you are talking about an HRT constraint in the first place. While you, of all of us, are more likely to run into an HRT constraint given the real-world applications which RDOS is used for, AFAICT none of the actual deployed installations to date have run operations with such a constraint (IIRC, they include things such as POS systems for gas/petrol stations, but don't control the fuel pumps themselves). So, unless your real application is something different from those you've mentioned, I'm not sure if this really is HRT.

As a rule of thumb, if it isn't controlling physical equipment, it isn't HRT.

(As an aside, I keep reading that as 'hormone replacement therapy', which says a lot about where my head is these days, but that isn't really relevant to this discussion.)

[LtG]

As a rule of thumb, if it isn't controlling physical equipment, it isn't HRT.

HRT simply means that missing a deadline constitutes a total system failure.

I think you kind of got it backwards, HRT has nothing to do with physical equipment or life, but rather when life is concerned total system failures are unacceptable and thus mandate HRT.

It's hard to say what constitutes total system failure though. Smart phones usually have dedicated processors for the mobile network, as failures would cause problems, but is it HRT? Failure shouldn't cause the phone to crash or break, but might cause the call to be dropped or disturbance for other users of the network. Given that calls are a primary function of phones, I think it's fair to call it HRT.

On a laptop I find it unacceptable for a single network packet to be dropped due to buffers being overrun, or audio issues because audio buffers were exhausted. I consider both to be RT.

I think both "total" and "system" are open to interpretation so I find it more useful to think about these from the perspective of the scheduler, does it guarantee (without exceptions) that deadlines are honored (HRT) or not (SRT), FRT is a subset of SRT.

And of course, the "without exceptions" is within context, if you use a hammer on the CPU then...

[rdos]

HRT simply means that missing a deadline constitutes a total system failure.

I think you kind of got it backwards, HRT has nothing to do with physical equipment or life, but rather when life is concerned total system failures are unacceptable and thus mandate HRT.

It's hard to say what constitutes total system failure though. Smart phones usually have dedicated processors for the mobile network, as failures would cause problems, but is it HRT? Failure shouldn't cause the phone to crash or break, but might cause the call to be dropped or disturbance for other users of the network. Given that calls are a primary function of phones, I think it's fair to call it HRT.

I kind of agree.

In my case, if I want to run continous analysis of the 1GHz data stream, it's clearly unacceptable to miss deadlines (buffers getting full) and it will cause complete failure of the system (FFTs and filters cannot handling missing data which will distort results). How tight the deadlines are depends on how large the buffers are. It's similar to audio streams getting exhausted, which will cause glitches in the sound. A similar issue is a network card buffer overflowing. This typically resuts in a need to restart the controller, which will lead to more losses. The losses themselves cause timeouts and resends, which clearly affects performance and how the system is perceived to function.

[rdos]

Not necessarily. Many modern hardware devices can operate in realtime and still send interrupts to the target operating system. A good example is a USB controller, a network card or an AHCI disc controller.

None of those are operations which usually have real-time constraints, not even soft ones. Online constraints, yes, but not real-time ones in a technical sense.

I didn't mean the data itself, rather the operation of the device. If the USB controller cannot handle the timing of USB data to or from device because it is waiting for the host, then data will become corrupt and performance will suffer. That is why even when PIC controllers have USB controllers, the USB controller has it's own state machine that is not dependent on the PIC processor itself. The same goes for network cards that must be able to send data in the correct way and cannot have this controlled by a more general purpose processor that also does other things. Network controllers, just like USB controllers, have their own state machines even when embedded in processors.

Generally speaking, RT is only going to apply to something where there is an output control signal being sent, rather than an input data signal being received. If the input is necessary for the system to decide how to proceed with a later RT operation, then yes, processing it may have a real time constraint, but even then that isn't necessarily the case. Even if it is, from the POV of the operating system, the constraint usually applies to the process (driver, application, etc.) which is creating the control output, rather than the input itself.

Well, if I develop some algorithm that outputs some important characteristics of the signal, whatever that might be, if the AD converter & algorithm cannot be operated in realtime without buffer overflows, then those algorithms will fail one way or another. The output itself doesn't even need to be realtime, as it might only consist of signals and timestamps when interesting things happened. However, this output couldn't be created unless the AD and software could handle the time constraints.

Also, a control output is only real-time if the operation must be performed within a fixed time window of no earlier than t and no later than t' to avoid unrecoverable failure of the operation itself - the lower time boundary is just as important as the upper one when discussing RT (even if it is defined as '0 nanoseconds', it is still a lower boundary), as is the requirement that a failure leads to some irrevocable outcome (which could be anything from a paper jam to a power plant meltdown, so long as it is something which cannot be ignored or undone).

Sure, but the constraint is that the AD converter has a limited buffer that must always be read before it overflows in order not to miss part of the data stream which might contain what you are looking for.

In other words, it doesn't sound as if you are talking about an HRT constraint in the first place. While you, of all of us, are more likely to run into an HRT constraint given the real-world applications which RDOS is used for, AFAICT none of the actual deployed installations to date have run operations with such a constraint (IIRC, they include things such as POS systems for gas/petrol stations, but don't control the fuel pumps themselves). So, unless your real application is something different from those you've mentioned, I'm not sure if this really is HRT.

I kind of agree. About the only example might be when you have dispensers that won't stop at limits so need to be sent stop signals by the controller.

OTOH, I've also done fuel dispenser software (using PIC controller & ARM processor), and that is typically done by regularly polling devices in a tight loop, which actually does provide a HRT solution since there are upper limits on how often you check the hardware.

[Korona]

My 2 cents: it does not matter if you declare a real-time problem to be hard or soft. I am not even sure if safety-critical applications care about the semantics here; they just care about meeting the deadlines. Yes, if missing a deadline means that people die, you have to try harder to meet it. But as a software developer you care about the tools that allow you to achieve this: regardless of the hard/soft wording, you want to design your system such that your algorithms have a constant worst-case running time and you want to avoid any source of spontaneous failure. This includes the failure to allocate memory after you entered the real-time part of your system. Hence, in that real-time part you just don't allocate but use other techniques such as intrusive data structures instead.

[rdos]

My 2 cents: it does not matter if you declare a real-time problem to be hard or soft. I am not even sure if safety-critical applications care about the semantics here; they just care about meeting the deadlines. Regardless of the wording, you want to design your system such that your algorithms have a constant worst-case running time and you want to avoid any source of spontaneous failure. This includes the failure to allocate memory after you entered the real-time part of your system. Hence, in that real-time part you just don't allocate but use other techniques such as intrusive data structures instead.

I'd say you pre-allocate everything. If I need a 100GB buffer I allocate it before I start the device. The other characterstic of a HRT device is that it will run in a tight loop that has bounded upper times on how long it will take to execute it. That's a requirement if you want to handle a continuous datastream without losses. You cannot have the OS scheduler switch to anything else as that might lead to buffer overflows.

[Korona]

Well, that depends on a lot of factors. You cannot switch to another thread for unbounded time but you might e.g. switch to another thread for 3ms if you know that your next action only needs to be taken in 5ms. In general, yes, you want some slack such that your deadlines can be met. You might want to look into real-time scheduling techniques such as earliest deadline first that allow the implementation of such systems. One thing to note is that real-time XOR fairness since whatever thread has the highest priority gets 100% of the CPU time.

[rdos]

Well, that depends on a lot of factors. You cannot switch to another thread for unbounded time but you might e.g. switch to another thread for 3ms if you know that your next action only needs to be taken in 5ms. In general, yes, you want some slack such that your deadlines can be met. You might want to look into real-time scheduling techniques such as earliest deadline first that allow the implementation of such systems. One thing to note is that real-time XOR fairness since whatever thread has the highest priority gets 100% of the CPU time.

Time constraints in the millisecond area should work in most mainstream operatingsystems (and certainly in RDOS by increasing thread priority). However, what I have in mind is to use the tight loop of typical HRT devices that can guarantee that time constraints in the microsecond or even nanosecond range can be met. That requires that you dedicate one or more cores in the processor for only this function, and that the OS scheduler will not operate on it at all. The communication with the host operating system can use memory based "schedules" and new data can be signalled with interrupts, much like how USB controllers and network cards operate.

[Korona]

Yes, that is a reasonable strategy. If you have deadlines in the microsecond range, you will want to dedicate a core. But doesn't your device offer buffering anyway? Surely, it cannot fill 100 GiB within milliseconds? (PCIe is obviously not fast enough for that.)