Hardware support for the hard real time

[AnishaKaul]

From here: http://forum.osdev.org/viewtopic.php?f= ... 47#p208546

What CPU are you targeting? If it's x86, forget about hard real time completely. SMM makes it impossible to implement a hard real time system.

Code: Select all

anisha@linux-dopx:~> uname -a && cat /etc/issue
Linux linux-dopx 3.2.1-12-desktop #1 SMP PREEMPT Thu Jan 19 16:02:50 IST 2012 x86_64 x86_64 x86_64 GNU/Linux
Welcome to openSUSE 11.3 "Teal" - Kernel \r (\l).

I again checked Xenomai's hardware support: http://www.xenomai.org/index.php/Embedd ... upport#x86

x86
All x86-compatible 32/64bit processor types starting from i486 CPUs and better are supported. Lower latencies are achieved when a TSC and local APIC are available from the CPU. Make sure the processor type you pick in your kernel configuration actually matches your CPU. In case you get weird latencies when running Xenomai on your x86 board, make sure to check out this information.

Can SMM not be disabled? How do they claim to support the x86 architecture then? RTAI also supports it.

[Combuster]

It's not as much the x86 processor that voids the possibility of true realtime - after all if you control the code in SMM then there's nothing wrong with that.

However, on the vast majority of x86 machines - the PCs and servers - the BIOS owns SMM space, not you. And the BIOS doesn't intend to let you get to it. So for any x86 equipment you're likely to get your hands on, hard real time is not an option.

[Brendan]

Hi,

Can SMM not be disabled?

SMM is present on all 80486 and later CPUs. There's no way to disable it. With specific code for specific chipsets you could attempt to prevent the causes of SMI, but that's a massive amount of hassle and likely to cause hardware damage anyway. Anyone who thinks disabling SMI is a good idea (e.g. Xenomai) shouldn't be trusted.

How do they claim to support the x86 architecture then? RTAI also supports it.

They never provide any guarantee about how long anything will take. If they did, what are you going to do if something takes longer than their guarantee anyway (ask for the $0 you paid to be refunded)?

The best guarantee they can provide is "things might not take longer than they did when you did your own benchmarks for whatever hardware you used". This shifts the problem of being unable to make guarantees to the person using their software.


Cheers,

Brendan

[AnishaKaul]

This SMM thing is quite enlightening for me.

http://en.wikipedia.org/wiki/System_Man ... tering_SMM

SMM is entered via the SMI (system management interrupt),

By design, the operating system cannot override or disable the SMI.

So, last time I configured the kernel, I found there an option to disable SMI.
[I don't know its side effects yet, will read on them separately]

So, if we disable SMI will that not disable SMM automatically? Or I have missed a point?
Wikipedia or Google doesn't seem to be talking much about SMI.

[Brendan]

Hi,

So, if we disable SMI will that not disable SMM automatically? Or I have missed a point?

SMI/SMM is typically used for a variety of purposes. These include:

• Emulating legacy devices that don't exist. An example of this is the well known "Legacy PS/2 emulation for USB keyboard/mouse" feature. For an OS with decent hardware support this cause of SMI/SMM isn't a problem at all, because the native driver would disable the emulation anyway.
• Power management. An example of this is fan speed control. For an OS with decent ACPI support this cause of SMI/SMM isn't a problem at all, because the native driver would use ACPI (and enabling ACPI for power managment disables the use of SMI/SMM for power management).
• Various fault handling and emergency purposes. This includes things like "memory scrubbing" for systems with ECC, the "hold the power button for 5 seconds" fail-safe used when the OS/kernel locks up badly, probably critical over-temperature problems (e.g. emergency fan speed control if the OS/ACPI failed to do it right or locked up), etc. These are the things that matter.

If you ignore the stuff that doesn't matter anyway, you're left with only 1 important question: Given the choice between missing a hard real time deadline slightly because of SMI/SMM and missing a hard real time deadline permanently because the computer caught fire and/or melted and/or crashed (because someone disabled SMI/SMM); which way would you prefer to miss the deadline?


Cheers,

Brendan

[AnishaKaul]

Firstly, again a thanks for great detailed post.

missing a hard real time deadline permanently because the computer caught fire and/or melted and/or crashed (because someone disabled SMI/SMM); which way would you prefer to miss the deadline?

Alright, well, <ahem> didn't know that disabling SMI
can actually cause fire or something. And I also hope
that you were not joking with that fire stuff!

Now, besides ACPI support what other properties should
I look in the computer to know whether it is robust enough
not to catch a fire on disabling SMM?

Not trying to strip the skin of a hair strand, I am asking
all this for a general information so that we can take an
appropriate decision w.r.t Xenomai and all that.

Also, does disabling SMI through kernel configurations
"actually" disable SMM?

[rdos]

Emulating legacy devices that don't exist. An example of this is the well known "Legacy PS/2 emulation for USB keyboard/mouse" feature. For an OS with decent hardware support this cause of SMI/SMM isn't a problem at all, because the native driver would disable the emulation anyway.

Uhm. But if the physical PS/2 connector exists, wouldn't that mean that the legacy device also exist, and thus that no SMM emulation would occur? Isn't this only a problem when you have an USB keyboard, and try to access it like a PS/2 device?

Also, what about HID support? If I decide I don't want to support the full HID specification, but rather leave keyboard in "boot emulation", that wouldn't involve SMM would it?

[Brendan]

Hi,

But if the physical PS/2 connector exists, wouldn't that mean that the legacy device also exist, and thus that no SMM emulation would occur? Isn't this only a problem when you have an USB keyboard, and try to access it like a PS/2 device?

If the PS/2 port and the PS/2 controller both exist but a PS/2 keyboard doesn't exist; then the PS/2 keyboard doesn't exist.

It's only a problem if you don't have a decent USB controller driver that disables the emulation. Even if there's no keyboard or mouse of any type, SMM might still be doing some emulation (e.g. trapping accesses to the PS/2 controller's IO ports).

Also, what about HID support? If I decide I don't want to support the full HID specification, but rather leave keyboard in "boot emulation", that wouldn't involve SMM would it?

The device's state (e.g. which protocol a USB keyboard uses, including the "Boot Protocol") has very little to do with the controller's state (e.g. if the USB controller has legacy keyboard mouse support enabled).


Cheers,

Brendan

[Brendan]

Hi,

missing a hard real time deadline permanently because the computer caught fire and/or melted and/or crashed (because someone disabled SMI/SMM); which way would you prefer to miss the deadline?

Alright, well, <ahem> didn't know that disabling SMI
can actually cause fire or something. And I also hope
that you were not joking with that fire stuff!

Now, besides ACPI support what other properties should
I look in the computer to know whether it is robust enough
not to catch a fire on disabling SMM?

If the SMM code does nothing (and is therefore safe to "disable"), then the SMM code does nothing and there's either no point disabling it or there's nothing to disable. If the SMM code does something (and is therefore not safe to "disable"), then you can't safely disable it.

Notice how there's no point disabling SMM/SMI in either of these cases...

Also, does disabling SMI through kernel configurations "actually" disable SMM?

No.

First, it's impossible to disable SMI/SMM in the CPU itself (you can only disable causes of SMI in the chipset). Second, the Linux kernel developers aren't incompetent - they'd know that there's never a good reason to disable SMM/SMI.

If you're seriously worried about SMI/SMM then you need to be able to control the firmware. For this purpose I'd recommend looking into the coreboot project.

However, I should point out that any system that actually does have hard real time requirements will be a critical system that also has stability constraints and will likely involve a risk to human life. In practice this rules out the use of anything obtained from anyone that can't be sued if the product they've supplied has design flaws; including coreboot, Linux and Xenomai (due to licence legalese) and anything supplied by Intel or AMD (as their legalese explicitly states that their products are not intended for "use in medical, life saving or life sustaining" applications).


Cheers,

Brendan

[gravaera]

Yo:

@OP: Basically, no generic or portable RTOS kernel solution will be taken seriously for real world hard realtime. A custom solution will always be better tailored to suit the specific needs of the machinery. Any client who needs an RTOS solution is manufacturing something that has a lot of money and reputation on the line. Anyone who has that tremendous amount of expectation to unfailingly deliver will prefer the most efficient solution for his particular product. The most efficient solution for an RTOS management package will always be one that takes on only the specific needs of the client's product and uses no generalizations.

Ergo, no such thing as an augmentation to a contemporary kernel to "support" hard realtime will ever get anywhere far.

Would you fly on a plane using a stripped down Linux with "RTOS patches" applied?

EDIT:

we can take an
appropriate decision w.r.t Xenomai and all that

Are you trying to investigate an off-the-shelf solution for a new product for a company or something? If you have pressures from your superiors or something, or your bosses don't understand what is involved, they can be referred here to for an explanation of why they shouldn't try to rush for an "easy" way to get a realtime system on the market.

EDIT2: You've also been implying you're working with someone else, or in a team. Out of curiosity, exactly who is we?

--Peace out
gravaera

[bluemoon]

How about running an OS within an isolated machine inside PCI processor board? There is less things to worry on SMI.
Unless you want something both fast and support realtime

[AnishaKaul]

Thanks everyone again for posting constructive details.

From wikipedia I have understood that SMI is an interrupt which
gets triggered when hardware gets (for example) heated up or
when a low level software wants to perform an I/O access to a
sensitive hardware like the motherboard etc. ?

So, setting that option [CONFIG_XENO_HW_SMI_WORKAROUND]
off, will ofcourse stop the software interrupts. Now, the question is - will
this stop the concerned hardware interrupts too?

From here: http://www.xenomai.org/index.php/FAQs#T ... x86_kernel

CONFIG_XENO_HW_SMI_WORKAROUND
This tries to disable System Management Interrupts on Intel chipsets. These interrupts run at higher priority than Xenomai core code or even the I-pipe layer. So they can cause high latencies to the real-time jobs (up to miliseconds). ATTENTION: On some systems, SMI may be involved in thermal throttling of the CPU. Thus, switching it off can cause hardware damage in overheat situations.

They seem to be saying that it can disable the hardware interrupts too!
They have written "some systems" on the page dedicated to x86 setups, so
Is the line in bold applied to all x86/64 processors? I feel the answer is
yes, but then not sure why have they specifically written "some"!

Actually, we have a software written in C++ which controls some unmanned systems
by obstacle avoidance algorithms, shortest path finding algos and all. So, currently
I don't know which kind of real time does my boss need, and he doesn't know much
about real time (currently). I'll give him a study of what works in which situation and
what doesn't, and will let him decide whatever he wants.

I work in robotics company. So, when I say "we" it means the senior authorities in the
company.

[gravaera]

Yo:

For the most part, if your machinery is to be used privately, and you don't intend to distribute, the problem is then mostly going to be possibility of business equipment failure -- that's something that will affect only your business, so all cool :O I had this fear that some company somewhere was trying to push out a product on the market "quick and easy", then market it as the best thing since sliced bread.

Don't forget: SMI isn't only used for managing failure, but to emulate devices, and even silently rectify board design anomallies by presenting say, the correct behaviour to the OS even if some device or other on the board isn't fully compliant with one or more standards. In other words, disabling SMI may or may not cause bugs which the firmware was masking to be exposed to the OS, and it may or may not cause certain devices to malfunction or behave inconsistently. It may or may not, for any particular board, make the board unusable.

Given that you're probably not going to test and implement on a large number of boards though, good luck, and I hope you find one that doesn't become unstable on disabling SMM/SMI, or even worse, seems to be stable, but has very subtle bugs that won't manifest themselves consistently. Firmware is responsible for booting CPUs, detecting faulty CPUs, initializing devices, applying microcode updates to CPUs, etc. The vendor knew more about the board than you do (E.g., "This is a low budget board, aimed at college students who need internet connectivity and low end graphics for word processing"), and they put the firmware there to circumvent whatever shortcomings the board had that prevented it from meeting its service expectations.

In short, unless you ask for a custom board to be made for your product, or you get a board's documentation from the vendor, the answer is generally: "You cannot predict the behaviour of a board when SMI is disabled, and should not rely on it producing stable behaviour for any mission critical application."

--Good luck
gravaera

[Brendan]

Hi,

So, setting that option [CONFIG_XENO_HW_SMI_WORKAROUND]
off, will ofcourse stop the software interrupts. Now, the question is - will
this stop the concerned hardware interrupts too?

Disabling causes of SMI in the chipset (e.g. the idiotic [CONFIG_XENO_HW_SMI_WORKAROUND] hack) won't effect software interrupts, exceptions, hardware interrupts, NMI or machine check (unless SMM is being used to prevent non-standard/undefined behaviour where disabling SMI/SMM could effect the reliable operation of almost anything).

From here: http://www.xenomai.org/index.php/FAQs#T ... x86_kernel

CONFIG_XENO_HW_SMI_WORKAROUND
This tries to disable System Management Interrupts on Intel chipsets. These interrupts run at higher priority than Xenomai core code or even the I-pipe layer. So they can cause high latencies to the real-time jobs (up to miliseconds). ATTENTION: On some systems, SMI may be involved in thermal throttling of the CPU. Thus, switching it off can cause hardware damage in overheat situations.

They seem to be saying that it can disable the hardware interrupts too!

They're only saying that turning off SMI could cause hardware damage. They didn't mention hardware interrupts anywhere in that paragraph.

They have written "some systems" on the page dedicated to x86 setups, so
Is the line in bold applied to all x86/64 processors? I feel the answer is
yes, but then not sure why have they specifically written "some"!

The unfortunate reality is that nobody (except the motherboard/firmware manufacturer) actually knows exactly what SMI/SMM is being used for in any motherboard. Even if you know all the details for every chip on the motherboard you can't know how those chips have been wired up to the rest of the motherboard or what the firmware might be doing. That's why only incompetent fools would consider disabling SMI/SMM. Also note that testing doesn't guarantee reliability - e.g. "I disabled SMI and everything seemed to work for 6 weeks!" does not imply that the there wasn't problems that weren't noticed or that there won't be a critical failure after 7 weeks.

I will also repeat what I've said before - if your system is important enough to require hard real time, then you shouldn't be using 80x86 in the first place. It's like a moron complaining that their car sinks to the bottom of the ocean, even though anyone with any intelligence would've used some sort of boat instead. It's not like hardware intended for the purpose doesn't exist (e.g. ARM, TTE systems, etc).

Actually, we have a software written in C++ which controls some unmanned systems
by obstacle avoidance algorithms, shortest path finding algos and all. So, currently
I don't know which kind of real time does my boss need, and he doesn't know much
about real time (currently). I'll give him a study of what works in which situation and
what doesn't, and will let him decide whatever he wants.

If the obstacles you're avoiding don't move, then you don't need hard real time or soft real time for this.

If the obstacles you're avoiding do move, then for safety reasons you need a conservative "margin for error". Accepted practice is a factor of 2 - e.g. under normal conditions, if your robot could move at a certain speed and still avoid all collision, then your robot will actually move at half that speed just to make sure. If you've got a conservative margin for error then you don't need hard real time. You probably only need "very soft real time", where "very soft" means that any OS should be good enough if no other processes are running, and an OS where high priority processes preempt lower priority processes should be good enough if other processes are running.


Cheers,

Brendan

[rdos]

I'm in the middle of designing a PCB that has requirements that cannot easily be met with a standard OS running on x86 hardware. Initially, I pondered on using a multicore x86 processor, letting one core run without interrupts and SMI, but given the extreme slowness of the dual core Intel Atom processor, I've changed my mind. The gate call performance benchmarks also show very different results between runs, even if I don't do power-management on it. Those taken together made me change my mind about implementing it that way.

So, now I instead integrated a Microchip PIC controller that will handle the realtime requirements much cheaper and predictably than a x86 processor ever could. This processor could poll ports 100,000s times per second and never miss a deadline. Of course, it won't use interrupts, but will do busy-polling. All instruction timings are predictable, so it is always possible to calculate response times.

BTW, the application is to decode clock and data pulses from a magnetic card pulled by the end user. It should work even if the user pulls it very fast, something that cannot be handled predictably on our current hardware based on x86.