About mouse handling of knobs & similar controls

FL Studio SDK and related information for developers. To ask questions please use FL Studio Looptalk

Return to “Developer Arena”

[You can only see part of this thread as you are not logged in to the forums]
gol
Thu Aug 07, 2008 5:44 pm

x

About mouse handling of knobs & similar controls

This is an article I once posted at KVR, can be interesting if you're making plugins.

Because everyone's plugin run in the same environment, because a user doesn't expect controls to be handled differently within the same OS, I'm posting an article / some thoughts & guidelines about knob & slider controls, that you're free to follow or comment.

Note that I'm discussing the handling & implementation of the motion, not graphics, as you can map pretty much anything to this handling system.

1. 'up/down' vs 'circle around'

Personally I prefer 'up/down', never found circling around a knob very intuitive, at the same time, I won't discuss tastes & colors..

However the circling around method has a valid point: it allows very accurate tweaking or fast tweaking depending on the distance between the cursor & the center of the control. This is pretty cool, but suffers from the 'near to screen edges' problem (more later), and its weakest point: the cursor starts at the center of the knob where it's very hard to tweak.
The circling around method could probably be improved by starting the tweak only when the cursor has reached a minimum distance, and also tweak the knob in relative motion/increments (helped by a visual overlay), rather than just set the position depending on the angle.

But I will rather discuss the most common method: the up/down one. It's nice because it also fits vertical sliders, numerical selectors, etc. It's very intuitive, everyone will probably understand it. I don't know who came with it first, I would say it appeared in ReBirth 338 (I think RubberDuck had circling-around knobs), but I don't know, maybe it's much older?

1.1. Hiding the mouse cursor

Hiding the mouse cursor when the control is clicked is quite common, and has several purposes:
-it makes the user feel like he grabbed the control/control's handle
-it hides the artefact of mouse position warping (more later)
-it hides the artefact of mouse position not linked to the slider's handle (when not using 1:1 mapping)

1.2. 'up/down' vs 'up/down+left/right'

Some had the 'good' idea to make the mouse horizontal motion also tweak the control, often in smaller increments. Not a bad idea in theory, sometimes well implemented, sometimes not.

Personally, I don't much like it, but if you implement this, please do it well: consider that no human can move his mouse cursor straight vertically, so you will have to handle the horizontal motion only after the user has passed some threshold. However, keep in mind that if you're also hiding the mouse cursor, it becomes difficult for the user to even know he's moving horizontally, which is why I would avoid this method.
You may also want to link horizontal motion to horizontal sliders, it's not so much more intuitive, in fact I prefer to handle such horizontal sliders traditionally (like scrollbars).

2. Speed/precision

Not everyone is using the same mouse speed/acceleration settings. It's pretty hard to decide how to map pixels to increments. If the mapping ratio is too high, you will lose a great deal of precision and the control will move too fast. If that ratio is too low, the user will have to slide his mouse for too long in order to cover the whole slider.
So here I would say: try to copy your favorite plugins, or the ones you think your users are already using, so that they will quickly get used to your controls. And stick to that average speed, only moving away from it a little if you feel like a control needs more precise tweaking.

3. Keyboard shortcuts

There seems to be several standards here. I guess they're all fine, but if you haven't decided yet, just stick to one of the existing standards, don't add a new one.
When I started making my controls, there were no such standard practice, so I'm not changing mine. I will detail it, and if you would like your plugins to behave nicely in FL Studio, you can follow it, however be aware that many VST's use different shortcuts.

Slow motion: CTRL+click, and right click (if the control has no popup menu). The slow motion factor is not a constant, it depends on the maximum precision your control can achieve.
Default: ALT+click. Just reset the control to its default value.
Bypass pauses/ticks: SHIFT+click. Because sometimes you want to tweak a control without feeling the pauses. You just hold shift while tweaking the control. Can be combined with slow motion.

Very often CTRL, SHIFT & ALT are swapped for the same features, or some of the features aren't present. And why not.. There's no better method, so just know which host & plugins your users are using, and make them feel at home.

4. Pauses/ticks

It's quite handy to feel pauses/ticks under your mouse cursor, to quickly snap to key value(s) (often that value is the default position, but it can be anything, like steps for a delay time selector, etc). It's not really a snap (it would suck), but a short pause, a larger part of the mouse motion spent on key values. You can even add visual indications when hitting those key values.

5. Scaling

Quite often you want to control something for which the stored running value is not in a linear scale (filter knobs for example). You may think that you could just build a logarithmic scale system right in the control itself. Don't! Bad idea. Your UI control won't be the only one controlling your parameter, external control (hardware) or automation will too, and the user expects the same scale as for the knob. So that scaling has to be performed in the parameter change itself.

6. Smoothing

Some controls have built-in smoothing, a nice example are the nice knobs in Ohm Force's plugins. It has pro's & con's. It's nice because the control feels nice, you get analog-like control instead of steppy increments. But for this you have to implement the smoothing in the plugin's processing itself, not the GUI (that normally is in a lower-priority thread, and you would rather have a smoothing that still works on high mixer CPU usage).

However, if you implement smoothing in the plugin's processing, you will not only smooth up user realtime control, but also automation control (I don't know if VST can differenciate both, but anyway it's probably not even possible to achieve, you could have automation with parameters being controlled live, so it would be both automation & realtime control at the same time). Automation should respond as quickly as possible, if the user wants a straight jump, that's what he should be getting.
So, if you implement smoothing, better make it optional. Personally I prefer to avoid smoothing in every control, but rather implement it in freely-mappable modulation knobs. That way the user can still control the source knobs precisely.
Finally, consider that the host can also have its own automation smoothing in its 'MIDI learn' system.

7. Programming

Programming those controls can be quite a problem. I will only write about implementation in Windows, because I don't (want to) know the Mac OS (feel free to add info).

7.1. Bad programming

See VSTGUI. Well, I hope/suppose it has been changed/fixed since then, but it really was bad programming. Windows is message-based, so your control should be too. Don't make clicking on your knob enter a loop like a noob, you will just prevent the application from getting its messages, and appear frozen. And no, plugin idling does not replace a message loop.

7.2. Methods & OS/input devices problems

So we already know that we want the mouse cursor to disappear when we click the knob, & track its vertical motion. Right, but what when your mouse cursor hits the screen edges? You're stuck & have to re-click on the knob to tweak it further. You may have seen this happen in poorly-coded plugins.

Side note about hiding the cursor: be sure to handle every possible way (like switching to another task) to leave your control. You don't want the mouse cursor to stay hidden because another window stold the focus while you were tweaking a knob. It happens.

When I first implemented knob controls, they were running in Win95. The solution here was quite simple: setting the mouse clipping rectangle to something much bigger than the screen, so that the mouse could get outside the screen without problem. However, this stopped working starting from Win2k, so we can forget this method. You can no longer set the mouse clipping rectangle to something bigger than the screen.

Some programmers chose to constantly relocate the mouse, optionally relocating it to the screen center at startup. This is bad for 2 reasons. First, it won't work with pencil (absolute) devices. Second, in the case the input device had some running accumulator, resetting its position may also reset its accumulator, so you would lose precision by constantly relocating it. I don't know if this happens in practice, but it's a possibility.

So the right solution is: leave the mouse where it is, track its motion, and when it gets close to the screen edges, you warp its position. When it gets close to the top, it reappears at the bottom, it's as simple.

Now, pencil devices. The problem with them is that they're absolute, you can't relocate the cursor (well, you can, but the next pencil motion will simply reset the mouse position - it's not like you could move the user's hand around).
The above method works with them, you will only face problems with controls that are near the screen borders. But as I don't see any other solution, this is the best we can get.
For better use with pencil devices, you can add optional support that would:
-ignore the mouse warping that will give troubles (controls near screen edges will still suck, but there's nothing we can do)
-reduce the motion to increment ratio. You may hide the cursor, but the user's pencil is still over the screen, and the user would probably prefer to have something closer to a 1:1 mapping, especially when tweaking sliders. This will also reduce the risk of hitting screen borders.

Vista introduced another problem, and another bug.
When you change the font size in Vista, it post-stretches the whole app windows (& it's pretty ugly, a bit like when setting an LCD monitor to a non-native resolution). But I see where they're getting at: while pretty ugly with non-integral ratios, I can imagine that one could buy a 2560*2048 monitor, and have perfect compatibility with existing apps thanks to a 2x stretching. Cool, but this new thing causes problems in our knob tweaking. The solution will involve the use of Vista's LogicalToPhysicalPoint & PhysicalToLogicalPoint functions. However, nothing will really work nicely due to a bug (in Vista or mouse drivers, I don't know), a simple GetCursorPos/SetCursorPos will behave wrongly when the window is stretched, placing the visible mouse cursor at a totally different position than where the mouse really is, quite bad. I guess that's what service packs are for.
The real solution is to ask Vista that you handle the scaling yourself (SetProcessDPIAware) (even if you don't).

Finally, when you release a knob, you show the cursor again. Ok, but now where is our mouse cursor? We could leave it where it is, but since the cursor was hidden, and since we warped the position, the user probably has no idea where his mouse cursor will pop up. So we relocate the mouse cursor to the center of the knob, or in the case of a slider, to the center of the slider's handle.
Only problem with this method is that 'mouse sliders' (users who 'slide their mouse around', as opposed to 'mouse lifters', who constantly lift & place back their mouse) will have to lift their mouse or quickly get out of their mouse handling area. But we can live with that.

Final thoughts:

There are options I haven't investigated. XP tablet version & Vista offer a virtual ink SDK, that tracks & buffers pencil (& probably mouse) motion very precisely (not just with sucky integer coordinates). Would probably be very efficient/precise/smooth for knob control, especially with pencils which are very accurate. But maybe a little overkill, and not compatible with standard XP.

Maybe more can be done using raw mouse input, if you have investigated this & got good results, please share. Or if you have a better method than the one listed above, I would like to read it.

Return to “Developer Arena”