GUI design thoughts, from both sides of the fence

[mystran]

I don't know if I'm mentioned this before, but there was a time when I wrote a small GUI toolkit for X11. In the process I learned quite a few things about GUI toolkits and event-based processing.

On the other side of the fence, I've since seen people misuse GUI APIs in ways which result in applications that generally work fine, but start lagging and "not responding" once there's enough things happening at the same time.

So, I've been thinking of writing an article or two about how to write GUI code that behaves well even on a slow CPU or when resources are running low, and that does stall because it needs to do IO or whatever.

But then the idea hit me, why not share some thoughts here as well, since while most OS probably aren't quite far enough that GUIs are really relevant, people here should be interested in systems programming, and maybe this whole mess will be helpful in some sense.

SO..

The absolute basic of a event-driven GUI ofcourse is the event-loop. That's some sort of a queue, where incoming events from whatever source (keyboard, mouse, windowing system, the application itself) get dumped, and then processed one at a time. Not much interesting in the loop, other than to mention, that there is one way to shoot yourself in the foot with the loop, and that's to write another special case loop. And the problem with special case loops is that you typically do that in order to not process all events in the normal fashion. And you want to always process any events, especially those you don't understand. Right? So the only exception I can see is modal dialogs, where you might want to drop user input (keyboard/mouse) going into the parent of the modal dialog, but even then you have to process any other events going into the parent (say redraws, internal messaging, ..).

Anyway, that's easy. The things I really want to talk about is stuff like drawing, which seems surprisingly hard to get right. Namely, every other programmer seems to think that when the application wants to draw, it asks the system for a handle to some graphics context, draws in there, and everything is fine. Which ofcourse is a seriously stupid thing to do in general, because drawing is typically expensive, and while drawing, you might get tons of other requests in the queue, which all require drawing, and your queue grows and grows and the feedback to user actions on screen lags more and more, and you either have to start dropping events or the whole thing crashes in an Out-Of-Memory condition.

The solution then, involves delaying the drawing in form of window invalidation. Which involves telling the system to send yourself a request to draw the window. Kinda. Except when you invalidate a window (or a region of a window) multiple times before actually handling the resulting request to draw, you won't get more than one such request, because multiple invalidations only make the window logically invalid once, and a single drawing operation is enough to make it valid again. Which means that as long as you can process any other messages faster than they arrive, you'll be able to clean up the queue after the drawing operations, independent of how long your drawing took.

And ofcourse you might then want to do other possibly expensive things related to drawing (complex layout calculations in response to window resizing come to mind) using a similar invalidation logic. With the same idea.

That's how a GUI can do very expensive drawing, yet behave acceptably on a 486 and Vesa framebuffer. Well, up to the point where drawing takes so long that it's essentially a synchronous operation and must be done in a separate thread with some double-buffering scheme (discussed below).

And then there those APIs which don't support the concept. Which means one has to build that on top of the API. And then people that don't understand the details will write apps on top of the API that freeze like it's not even funny when you trigger too much drawing too fast.

So mm.. if you design a GUI API, provide for proper invalidation of windows, and PLEASE PLEASE make it as hard as possible (IMHO impossible) to get a raw drawing context outside a repaint event handler. Thank you.

The other thing, that I wanted to talk about, is the fundamental incompability with an event-based GUI and synchronous IO and how the Windows Explorer in Vista freezes in response to SMB operations all the time. Not permanently, but until it timeouts or gets the operation done. Not nice.

So, mm.. you don't do synchronous IO in the GUI processing thread, ever. You shouldn't anyway. You schedule a background threads, tell it what you need done, it goes doing the stuff, sends an event notifying you that it finished (or failed) and you normal GUI event processing notices that such and such thing happened and acts accordingly. That way while the synchronous process blocks, the GUI thread can keep doing things like redrawing invalidated regions of the window, and waiting for the user to press "cancel".

And the reason I mention this, is because all too often one sees stupid APIs and make it totally impossible to send a result-nofication-event from the worker thread into the normal GUI threads event-queue. That's like.. a joke. No really, please, in an API design, make sure that the GUI thread never needs to block on anything. Some mutex locking for O(1) critical sections is totally fine, but just no unbounded waiting. It's impossible to design well behaving GUI apps then.

As for the user hitting cancel, well, sometimes the background thread could still keep blocking, and nobody would be any wiser, but in operations like transferring long files over the wire, it's trivial to check some cancel indicator ever once in a while to see if there's any point to keep going.. But that's ofcourse normal multi-threaded programming, and really hard to get right, so mm.. I'll not start discussing that now.

..

Was there something else? If you can think of other ways to shoot oneself in the foot with event-based GUI programming, please share your thoughts. If you disagree, share that as well, and I'll tell you that you are wrong.

Thank you.

[Candy]

The absolute basic of a event-driven GUI ofcourse is the event-loop. That's some sort of a queue, where incoming events from whatever source (keyboard, mouse, windowing system, the application itself) get dumped, and then processed one at a time.

That's the quickest I've ever disagreed with you. I intend to make a UI system (working on it as we speak) that processes them in parallel as much as possible; in any case not one at a time.

Not much interesting in the loop, other than to mention, that there is one way to shoot yourself in the foot with the loop, and that's to write another special case loop. And the problem with special case loops is that you typically do that in order to not process all events in the normal fashion. And you want to always process any events, especially those you don't understand. Right? So the only exception I can see is modal dialogs, where you might want to drop user input (keyboard/mouse) going into the parent of the modal dialog, but even then you have to process any other events going into the parent (say redraws, internal messaging, ..).

Seen that screwed up occasionally which lead to the application stack overflowing if there was enough network traffic (one every so often a packet handling routine would end up starting another message pump).

So mm.. if you design a GUI API, provide for proper invalidation of windows, and PLEASE PLEASE make it as hard as possible (IMHO impossible) to get a raw drawing context outside a repaint event handler. Thank you.

While you're at it, add local invalidate requests too, where you can signal that only a part is invalidated. Oh, and fill the buffer with random noise on debug builds (somehow) so users will learn to write the full buffer instead of hoping "it should still be what we wrote there last".

Was there something else? If you can think of other ways to shoot oneself in the foot with event-based GUI programming, please share your thoughts. If you disagree, share that as well, and I'll tell you that you are wrong.

I see you're open to discussion

[Craze Frog]

The absolute basic of a event-driven GUI ofcourse is the event-loop. That's some sort of a queue, where incoming events from whatever source (keyboard, mouse, windowing system, the application itself) get dumped, and then processed one at a time.

That's the quickest I've ever disagreed with you. I intend to make a UI system (working on it as we speak) that processes them in parallel as much as possible; in any case not one at a time.

That's of course completely nonsense and also plainly WRONG. Let's say your UI has a text field. Should it process keyboard event asyncronous? No, because then what you type will end up in the wrong order. This also applies to other things. Let's say the text field has focus and I click a menu item and then press a key before the UI system gets around to show the menu item. Users will here expect that the key press is not processed until after the menu is shown, because the menu should be shown when they pressed the mouse button and they pressed the key after that.

Not much interesting in the loop, other than to mention, that there is one way to shoot yourself in the foot with the loop, and that's to write another special case loop. And the problem with special case loops is that you typically do that in order to not process all events in the normal fashion. And you want to always process any events, especially those you don't understand. Right? So the only exception I can see is modal dialogs, where you might want to drop user input (keyboard/mouse) going into the parent of the modal dialog, but even then you have to process any other events going into the parent (say redraws, internal messaging, ..).

You don't want a separate loop for a modal window. Use no more than one UI loop per thread. Multiple loops is the cause of endless problems for beginning programmers who wants to make a popup dialog.

So mm.. if you design a GUI API, provide for proper invalidation of windows, and PLEASE PLEASE make it as hard as possible (IMHO impossible) to get a raw drawing context outside a repaint event handler. Thank you.

Of course it depends totally on what you want, but in MY very own personal nirvana GUI toolkit, the toolkit takes care of drawing, and it's simply not possible to draw from the application. Instead the GUI toolkit should provide an image widget. But that's my way. Of course, implementing the redrawing inside of the GUI toolkit then becomes another matter, but it's a whole lot easier when you know you know no one else is drawing on YOUR surface.

Was there something else? If you can think of other ways to shoot oneself in the foot with event-based GUI programming, please share your thoughts. If you disagree, share that as well, and I'll tell you that you are wrong.

One thing that's so easy to get right, yet every idiot gets wrong is saving the window size and state. If the window is maximized, that must be saved and then the position and size should not be updated, so that the restored geometry remains stored.

[mystran]

The absolute basic of a event-driven GUI ofcourse is the event-loop. That's some sort of a queue, where incoming events from whatever source (keyboard, mouse, windowing system, the application itself) get dumped, and then processed one at a time.

That's the quickest I've ever disagreed with you. I intend to make a UI system (working on it as we speak) that processes them in parallel as much as possible; in any case not one at a time.

That's of course completely nonsense and also plainly WRONG. Let's say your UI has a text field. Should it process keyboard event asyncronous? No, because then what you type will end up in the wrong order. This also applies to other things. Let's say the text field has focus and I click a menu item and then press a key before the UI system gets around to show the menu item. Users will here expect that the key press is not processed until after the menu is shown, because the menu should be shown when they pressed the mouse button and they pressed the key after that.

Well, there's actually a middle-road, which is surprisingly simple, and takes care of basic locking as well: color the events and maintain strict ordering for events with the same color. Any two events with differing colors can then be run in either order, or even in parallel. Now, as long as the coloring is documented properly, and apps programmers can control coloring of their own events, you can run the same loop/queue from multiple threads, yet avoid most of the nasty pitfalls.

There might be other solutions as well...

You don't want a separate loop for a modal window. Use no more than one UI loop per thread. Multiple loops is the cause of endless problems for beginning programmers who wants to make a popup dialog.

Ok, that's like one of the reasons I kinda posted this here. I guess you are right. I guess it's just impossible to get those right anyway, unless you are the designer of the API/Toolkit, and in that case one can design for other methods anyway, so let's make the official recommendation: No multiple loops for whatsoever excuse.

Of course it depends totally on what you want, but in MY very own personal nirvana GUI toolkit, the toolkit takes care of drawing, and it's simply not possible to draw from the application. Instead the GUI toolkit should provide an image widget. But that's my way.

But if the application needs to manually draw stuff anyway, then the best you can get out of this, is making the application draw in some sort of double buffer, and then have your widget dump that to screen when necessary. I mean, there are times when applications will want to control per-pixel output, whether or not you like it (prototypical example would be a raster graphics editor, like Photoshop). Oh, and then you could add a third buffer to where the application draws while you service invalidations from the main buffer to screen, and you end up with tripple-buffering, which is also quite fine, but not always worth the overhead. YMMV ofcourse.

One thing that's so easy to get right, yet every idiot gets wrong is saving the window size and state. If the window is maximized, that must be saved and then the position and size should not be updated, so that the restored geometry remains stored.

Ah, great point. Actually the proper solution is to store two pieces of information: the size and position of the window in restored (non-maximized) state, and then a boolean of whether the window was in maximized state when closed. That way the window can come back as maximized, yet when restored, it'll remember the most recent restored configuration.

[mystran]

Let's say the text field has focus and I click a menu item and then press a key before the UI system gets around to show the menu item. Users will here expect that the key press is not processed until after the menu is shown, because the menu should be shown when they pressed the mouse button and they pressed the key after that.

Actually, whether the keypress or the visual feedback (drawing the menu) is handled first would be completely irrelevant. Besides, you might not even be able to process the mouse action before the keypress is already in the queue as well, and you definitely shouldn't ever draw from the mouse event handler. Giving the draw request more priority than the keypress would then be equivalent to drawing in the mouse action handler, and equally stupid. But I guess you mean the mouse event must be handled first, such that focus is given to the menu, whether or not it actually gets drawn before the keypress is handled.

[distantvoices]

regarding modal windows (popups and sorta): The UI-Toolkit-designer has the nice possibility to deal with them in the exactly same event looper which takes care for the whole application. Simply have the loop check whether there's a modal window open - that's one static global field in the library or in any kind of application structure - a pointer to the current modal window - and here you go. Just take care to set that indicator back to null upon closing the modal window.

In a former version of my ui toolkit, I've used a loop per modal window - oh, what a crap that's been. I've caught the most inconvenient bugs with that model, so away with it.

Regarding syncronous io and event based ui handling: I agree: The file reading/writing should be definitely delegated to some thread of its own which watches over the process while the ui event looper can stroll along doing it's own stuff.

just my 2 c.

[Candy]

That's of course completely nonsense and also plainly WRONG. Let's say your UI has a text field. Should it process keyboard event asyncronous? No, because then what you type will end up in the wrong order. This also applies to other things. Let's say the text field has focus and I click a menu item and then press a key before the UI system gets around to show the menu item. Users will here expect that the key press is not processed until after the menu is shown, because the menu should be shown when they pressed the mouse button and they pressed the key after that.

Should your UI render all display bits sequentially? Why? Should it update the display after processing the full impact of your action, or should it invoke the action in parallel with the display invalidate (and potential update) ?

I didn't say it should just do any event at random.

[Craze Frog]

Let's say the text field has focus and I click a menu item and then press a key before the UI system gets around to show the menu item. Users will here expect that the key press is not processed until after the menu is shown, because the menu should be shown when they pressed the mouse button and they pressed the key after that.

Actually, whether the keypress or the visual feedback (drawing the menu) is handled first would be completely irrelevant. Besides, you might not even be able to process the mouse action before the keypress is already in the queue as well, and you definitely shouldn't ever draw from the mouse event handler. Giving the draw request more priority than the keypress would then be equivalent to drawing in the mouse action handler, and equally stupid. But I guess you mean the mouse event must be handled first, such that focus is given to the menu, whether or not it actually gets drawn before the keypress is handled.

Yes, but when the keypress is processed it should be handled as if the menu was already shown, so that the press goes to selecting a menu item rather than typing into the edit field. Forget actually drawing the menu.


About the colouring of events: I can't think of ANY UI events that can be done out of order without messing some things up.

[Candy]

About the colouring of events: I can't think of ANY UI events that can be done out of order without messing some things up.

I can draw the menu after processing your keypress in that menu.

[Craze Frog]

About the colouring of events: I can't think of ANY UI events that can be done out of order without messing some things up.

I can draw the menu after processing your keypress in that menu.

Drawing the menu should not have anything to do with handling the events.

So it must go like this:
Handle mouse event: ShowMenu()
Handle key press

You can't handle the key press before you've called ShowMenu(), but ShowMenu() can put the actual drawing of the menu in a thread so that the mouse event is handled quicker.

[Avarok]

I understood that UI input device activity caused an interrupt which resulted in the queuing of the device input. This accomplished, it then makes sense to run UI output based on a parallel thread to the actual processing of the input queue.

The event mechanism should be used and you should most definitely not be polling for new input in a loop.

That said, I also firmly believe that the UI output (display) should be handled by a completely separate shell "process" operating on shared pages. Programs should only ever post data to that shared memory to get it displayed.

This makes sense because
1) graphics libraries are implemented the same across applications on a per-shell basis
2) you do not want to leave application code messing around with "redraw events"
3) you desync application stability with GUI responsiveness
4) the GUI should reach a ready state only every display refresh, not every time an application gets CPU time.
5) it's simpler for app developers to program to a framebuffer than to an immutable API.

Best wishes,

[Candy]

About the colouring of events: I can't think of ANY UI events that can be done out of order without messing some things up.

I can draw the menu after processing your keypress in that menu.

Drawing the menu should not have anything to do with handling the events.

So it must go like this:
Handle mouse event: ShowMenu()
Handle key press

You can't handle the key press before you've called ShowMenu(), but ShowMenu() can put the actual drawing of the menu in a thread so that the mouse event is handled quicker.

Code: Select all

void mouseEvent(const MouseEvent &e)
{
    setMenuStateLogic();
    if (logicCausingARedraw()) {
        defer(ShowMenu, this);
    }
}

void keyboardEvent(const KeyboardEvent &e)
{
    if (menuActive) {
        menuKeypress(e);
    }
}

menuKeypress(const KeyboardEvent &e) {
    switch(e.key) {
    case 'x': exit(0);
    case 'o': menuActive = false; defer(HideMenu, this); break;
    }
}

Like so?

[Craze Frog]

Like so?

Yes.

[mystran]

Random ideas that I just got, but haven't really considered the implications of yet:

It's relatively easy to prevent applications from creating extra event-loops and from drawing outside the repaint handlers, by API design. But can't this be extended to other stuff as well...

...like let threads have a tasktype attribute, which is set when the thread is created, and then restrict different APIs to different tasktypes. Any single thread can only have single tasktype, so you can't put stuff that should be in different threads into the same thread.

Namely, restrict all GUI operations into a task-type "GUI". Restrict all audio operations into a task-type "multi-media" (which is also allowed access to streaming video overlays?). Then restrict all blocking IO into a tasktype "IO."

There would be two benefits actually: nobody would be able to do IO from a GUI thread, or audio from an IO thread. But even more importantly, the system scheduler could make more intelligent decisions depending on types of thread functionality.. and restricting APIs (forcibly) to certain task-types would force application writers to provide this information..

Such an evil plan...

[exkor]

I can draw the menu after processing your keypress in that menu.

So is it like in WIndows then? Press Start->Shutdown Enter button on keyboard -> computers reboots/shuts down without showing ShutDown Dialog.

And one more though. To ensure that gui runs in separate thread and no programmers code can get into that thread(to slow it down or froze) GUI has to be put in a text file(like HTML) and parsed during runtime or if you wish during compilation into the code that will only have GUI related function.