Designing a menu logic for a haptic UI model

Enhancements
1

Hi there,

I am tinkering on a logic for a model to build an intuitive-as-it-can-be control interface for WLED based lamps. My goal is to use a single push-button wheel (the thing a mouse has) to create a highly accessible and easy to understand interface that neither lacks in (necessary) options nor is too complicated to be controlled by our grandparents. :wink:

If we can build an interface that suffices Usability and UX requirements with such a minimal UI where input is limited to a push-button and output (feedback) is nearly non existent, then we can extend it with additional input (buttons, interfaces etc) and output (information, feedback etc) options.

Feedback Types

As a rule of thumb, the more and the better the feedback, the easier it is to build a good and easy to understand UI. But I want to take advantage of a minimal (or even non-) feedback interface. But lets take a look at possible feedback types here:

  1. On state change
    –> you push a button, sth. Happens. Most common form. Works fine for command executing, but not for moving within a hidden menu

  2. (Preview) LEDs (Single/Set/Ring of ws2812b LEDs)
    –> Even a single RGB LED (or even a single color LED) can be a great source of information. They could be an index for the actual menu location, and with only a few LEDs most of the effects could be outputted adequately before you chose one to apply.

  3. The lighting LEDs themselves
    –> Parts of the strip or (as in mine case) the matrix function as a „screen“. This would be a great resource for a menu, but as a downside depending on the diffuser you use it can be quite fuzzy and would probably require some tinkering to harmonize it for different matrices, strips and other forms. Could work as color and effect preview, but hardly for indexing menu locations.

  4. A dedicated display
    –> Would be the most straight forward option, and there are already one or some usermods taking advantage of this with an ESP by TTGO. But besides the additional costs (not important for one light, but for the production of a series) it limits the design of a generic controller (that development is also an important part of my work) pretty much (e.g. having an 2,5 cm wide TTGO ESP with a display built in would exceed the aimed 2 cm height of the controller box). Another disadvantage is the additional power consumption, as the first prototype I want to get in MVP status is a mobile light running on an internal battery or a common powerbank)

  5. Sound
    I havent thought it through yet, but a simple buzzer could also be used as a part of menu navigation/orientation or state-change-feedback. The crux here would lay in finding a matching code (i.e. what beep means what) that wont lead to irritation instead of making things clearer.

Control Use-Cases: State-Change vs. Search vs. Play

To better understand what a Light UI is used for, it can be helpful to refine the reasons why it is used.

Specific state-change
The most common type. There is a defined goal someone is trying to achieve: Increasing brightness, Choosing an effect, decreasing effect speed and so on. As no one wants to cycle through dozens (or hundreds) of options there should be a short pathway to reach everything within a couple of clicks. It may be tempting to subdivide everything into a branching tree structure, but even with a clear navigational feedback (knowing at what menu point one currently is) it requires classificational conditions like exhaustion and disjunction (or selectivity in simpler words). Every effect has a brightness level, but only most effects have an effect speed (not the static ones). All effects have a color (in some kind), but some have a single color, some have two colors, some have color palettes. This to dissolve isnt trivial, the biggest „chaos“ indeed is caused by the „Intensity“ parameter. Some examples what it can mean: Nothing, Smoothness, Duration, Number, Pastel/Saturated colors, size, Wavelength, width, fade rate, density, frequency, speed… This is a UI/UX designers nemesis :-X
Some of you may think here „so what?“ To be in control of something implies to know what parameter one is actual changing/manipulating before(!) doing this (keyword: conformity to expectations) This can either be reached by clear information (e.g. on a display) or by a mental model in the users head that fits the actual UI logic. But to reach this, the UI composition has to follow clear rules, otherwise it would require a huge amount of memory to not irritate the user and to avoid mistakes.
This gets even more confusing when the outcome of a state-change is of delayed or fine-grained nature („Did I just change anything, and if I did, what did I change?“).

I have no clear solution how to tame the Intensity chaos here, some considerations (that require some more research) that crossed my mind until this point:

  • Make it some kind of a fuzzy-magic-function that simply acts as an unclear effect modifier. But here also all effects should have such a function, not only a part.
  • As a variation of this, it could be hidden within a random button/function. Given a specific effect, a „magic“ button changes colors, speed and (if available) intensity randomly within all possible options or within a pre-selected subset of options.
  • Delete the Intensity modifier from the haptic UI. Could still be accessed within the app, nevertheless a pretty hard trade-off.

Searching
Sometimes a user is looking for something familiar that she can enclose more or less. This may be a saved preset, an effect or a color (palette). Most of you may know this „Wait, there was this one effect/preset that fits into this moment“ situation. „We know it when we see it“ can also describe this. In this case we have to cycle through all options to brute-force-find our wanted goal. Sometimes we have a clue about the position of our goal (e.g. one of the first presets, somewhere within the last third of the effect list, etc), sometimes not. Going through a list of 16 presets seems quite fair to me, but these 120+ (and still growing) effects are way too much if on a single list.
How can we reduce this number?

  1. Hardcut some of them out. In my case I would deliberately choose effects that doesnt fit proper to my built lights.
  2. Softcut them out. Meaning they are not available within the PushbuttonUI, but from within the app. Or they are limited within the Kiosk-Control mode with preselections, and can be added to it via a configuration mode.
  3. Classify them to make a branching tree via subdivision. This is a tricky one… There are (some) static and (most/many) animated effects. There are (some) sound-reactive and (most/many) non-sound-reactive effects. There are (some) static, (some) sound-reactive animated and (most/many) non-reactive animated effects). Also (not so clear): There are ambient and fast-paced effects. And: there are effects that have one color, two colors, or specific color palettes. There are (many) moving effects, (some) fading effects, (some) static effects and (some) XXX effects? Ah, and of course there are effects with an intensity modifier and there are effects without one. :wink:
    Quiz question: What are clear categories (and maybe sub-categories) we can divide the effects into?

Playing
Another motivational use. In most use cases we change a parameter or some; more or less randomly until they fit our situation. A common way would be: Choose an effect, choose a color, choose the speed, choose (if available) the „intensity“. Normally we would either save the result as a preset or just go back to our preset-main menu. There are several ways to handle the save function, especially regarding the kiosk-control mode vs backend-config mode question. My intuitive approach would be: Besides X presets there is one „free“ slot at the end of the list, so while cycling through the presets from left to right the user doesnt get annoyed/irritated by the placeholder light (thats of cause our lovely static ambient orange´esk one). But from the starting point just one click/move to the left one can accessed the free slot. As a general rule I would suggest: Long Press while in a menu of a preset –> Save and exit to the start (or the saved preset). Otherwise one is just clicking out of the menu without any state-change. Alternatively normal clicks just lead to cycle within a preset menu, and only a short double press exits it without state-change. Additionally, after exiting the placeholder menu with a save, a new free save slot is created next to it.

Bottom Line

So narrowing my thoughts down, I would conclude with these questions

  1. What you think may be good (exhaustive/disjunctive) classifiers for effects? E.g.: Static, Smooth/Flowing, Running/Moving, Blinking. Question here always are the same: Does it subsume ALL effects? Is every effect only subsumed under ONE classifier?

  2. How can we get rid of the intensity chaos? One possibility could be correlating them to another classifier, e.g. all effects that has their intensity slider making X are also all effects that may be classified by effect class Y

  3. As I like to have a display optionally installed I wonder if some guys already built a menu for navigation with one (A display should support navigation/feedback, but navigation mustn’t completely rely on a display)

  4. Attached a first model of a minimal menu logic that should be consistent and intuitive. Thoughts and ideas how to further improve this?

Thanks guys
Kelvin

ui
ui870×701 55.5 KB

2

Great to read about someone else thinking about this! I’ve been thinking about this for many years, and started thinking about it in relation to WLED last year.
You can read my musings of that in this chat (posting as emardee): Rotary Switch support ¡ Issue #334 ¡ Aircoookie/WLED ¡ GitHub

Specifically I was thinking about using a rotary encoder with a push-button centre as a simple interface… Which actually enables quite a few ways to configure a simple menu system. In my case I was planning to mount these to look like a traditional dimmer knob on a blank faceplate on tge wall. And make them behave like a basic dimmer by default… but with more features available for those in the know, (although I outline 3 different ways the menu structure could operate in that other discussion). I reckon simple feedback of where you are in the menu could be provided by some leds around the encoder (even behind the blanking plate even, shining through!).

As I mused in that discussion, I thought that ultimately WLED could probably support all the menu modes and rotary encoder, potentially with the preferred menu system and options being chosen by the user in the webgui.

Happy to talk more about this, and having someone to bounce the ideas off is likely to make something happen quicker at my end too!

I’ve made no further progress myself… because, you know… “life”!

3

Huhu @emardee,

having someone to bounce the ideas off is likely to make something happen quicker at my end too!

Yeah, great to see some potential sparing partners out there, so I directly dive into some thoughts.

First, when talking about a rotary encoder with knob, you are also referring to an “endless wheel” version (meaning you have a circle and not a line for moving)? I dont see any advantages but only limitations with the latter one.

I outline 3 different ways

In general the wheel should be “reserved” for anything with many options/states (lists etc.), as ways should be as short as possible for any distance one has to overcome in navigation. Also, the use of both UI elements, wheel and knob, should be as consistent as possible (meaning they should do more or less the same at every menu point. In my case the wheel is used for lists of presets, effects, colors and speed steps).
I think my model should be a bit different from yours, as your Tier 1 is my Tier 2 with having the presets on Tier 1. I would call it different design philosophies/approaches and could be researched within some later stages of user research (I started working on a research design for that, too).

Ah, I just see I didnt take brightness into account here (wrote the text a while ago). I think I was unsure if there should be some master brightness or it should be defined/saved for every preset itself. I had the idea to make “hold knob and cycle to l/r” change brightness for the current effect while just cycling makes the saved preset change (so thats kind of a very minimal kiosk mode while pressing the knob one time gets you into config mode). But that only works if there are not too many brightness steps or an acceleration is being used.

One menu element you could get some inconsistencies with is the “rainbow hue”. As I wrote there is some amount of chaos, as rainbow hue only works for effects with one color (i.e. “solid”). I see this as a crucial challenge, as I always have problems explaining the use of changing colors/effects. I say sth. like “some effects have their own color palette, but to see this you have to check if palette is on “default” and no colors are set within the color tripple. And if you change an effect and see nothing, then you should check if a color or a palette is chosen”. From my UI designers perspective these little things are crucial in the end bcs. irritation is highly annoying (conformity to expectations).

One possible solution crossing my mind: You can set primary, secondary and tertiary color, which are automatically saved as palette. Also you can set up up to X color favorites (with the rainbow hue).
When choosing an effect with palettes, the first one is default (if there is one), second (or first, if there is no default) is the “set colors” with the other palettes following. Similar for effects based on colors, where there is a list of your colors used instead of the palette list. Nevertheless it keeps being complicated, as there is mostly a secondary and rarely a third color being used. But my goal here is to find sth. like a canonical/normal form (mathematically speaking) to harmonize all this.

I have to go, so just a few further short thoughts for the moment:

  • As I can see you have thought about what I call a “preview feedback LED” as feedback element. We can take this into account as it simplifies stuff. But also here I´d say that the success of the UI navigation shouldnt completely rely on this. A haptic interface should imo always work when controlled blindly.
  • Besides the wheel-knob you got into and my pushbutton wheel I have thought about the use of a foot pedal. There are one with also a wheel, so in later stages could be 3 different UI elements with different purposes based on the same menu logic.
  • As mentioned I was thinking of creating a small user research study, as there are some simple usability methods (e.g. thinking aloud, card sorting, paper prototyping etc) that could give us some great intel about the users mind sets. If you like you are invited to support this, as it also helps us getting a better understanding of relevant questions and problems we want to solve (e.g. it could turn out that 90% of chosen favorite effects are concentrating on 20 or so effects)

Looking forward to your thoughts about all this.
Kelvin

4

I suspect the big difference from what you are hoping to achieve compared to what I’m aiming for, are the different use-cases of the lights, which might diverge our implementations into different directions. For me it is mostly for static solid colour. They spend 90% of their life in plain white. With some dimming or warming of the white colour at night. Being able to add full rainbow and other features from the dimmer greatly adds to the feel that can be created. This is probably much easier to achieve than the more complex settings you are hoping for.

But there will be still good learning and cross-polination from the generic stuff applicable to both our sets of needs… and wider than that… to other use-cases different from both our needs that other users may dream up! Down the line, I suspect this could be catered for with a generalised support of rotary encoders to carry out various functions and menu pages with customisation that can be made to suit the users specific needs from the GUI. In the short-term, like you say, it’ll be setting up various test scenarios and refining them.

In my case, I am less focussed on flashing patterns… (as I’m happy to use the phone app for those)… and more concerned about making the basics of operation available without needing to reach for a phone. I might include some patterns (especially a few good sound-reactive ones!)… but mostly for me, it will be used for static, whole string colour mood changes, and brightness changes. (pleased to see WLED has added a warm white to cool white slider in the last year… was planning to fake that myself, but can presumably just set the value for that existing variable now!). In the kids’ bedrooms, being able to set nightlight and “sunrise alarm” modes would be good features for my implementation too. I would envisage being able to select some preferred saved colours or even pattern presets too, but a minimal number. Will always need a full-on disco mode though to unleash the full capabilities! :star_struck: :smiley:

I see value in being able to use things like using short press… medium press… and long press of buttons which might enable advanced hidden modes that I would expect to stay hidden from the basic user. In fact, because I expect it to be intuative and remain simple, for that reason, I would expect that after not being touched for say 15secs, regardless of where you are in the menu, I am planning to revert back to a simple dimmer mode… so the next person to the panel gets what would be expected from a dimmer switch in a room… and it would operate as expected without any training!

In terms of hardware, yes, the rotary encoders I’m thinking of are infinite turn… so you can always go from where you are to max or min when changing to different values in different menus. Something like these:
https://nz.mouser.com/Electromechanical/Encoders/_/N-39xfc?P=1yyaxjxZ1yyawv7Z1yyaxf6Z1yyaxf7Z1yyaxfmZ1ywv286Z1y81857Z1y92fshZ1z0yz89Z1yzxmvnZ1yzusoeZ1yzxmvkZ1yzxmuw&Ns=Pricing|0

Centre-push button on the knob to operate a momentary switch. Currently considering differences between those with detents and those without… the detent giving a definite haptic feedback of the minimum change of value (assuming detent ppr matches the resolution - not all do). My wife favours no detents, but I find it too imprecise!

I might use acceleration in the code, such that a fast turn jumps up/down by say 10, 5 or 2 and slow turns increment by just 1. But I think that would be relatively easy to tune once the basics are working. Looks like the most cost effective encoders are the Bourne ones with 24ppr which seems to be a reasonable resolution without giving too many or too few changes per turn. But I’ll know soon enough if that is right. If I need no acceleration, I can change the increment step manually to tune it… maybe +/- 5 or 10 will work better than +/- 1 for use with no acceleration. Only testing will reveal that. I think you would want to be able to sweep the full range in about 1 full turn. But wouldn’t want too big a jump at the low end. I presume say for brightness, changing from 1 to 2, and 3 to 4, would be far bigger a change than changing from 253 to 254, and 254 to 255. So another option is to use 1s low down, then 2s, then 5s, then 10s as you sweep up the range. but WLED might already allow for better proprtionality in the code across the range. Something to check.

I was wondering about whether to add a reed switch (or two) behind the faceplate, so that other advanced modes could be triggered with a magnet in the right place (again for those in the know!). Probably won’t do that. But it is interesting to think of ways to keep it very simple and intuitive (and more importantly looking simple!), for most users… but allow for more advanced usage for someone trained! (Might still enable an advanced mode with more options when using a long press to switch modes).

For my use case, the led feedback becomes simpler, as colours, or brightness, or warmth can be potrayed using a few leds, to give the idea of what will happen if you rotate left or right at any given point (either using animation or just static indications)… therefore knowing which mode you are in… dimmer, white balance, rainbow, etc. I think the more menus and modes you have the more likely you are to be served better by a screen of some sort… unless the menu is very user friendly! Being for mostly solid colour changes for me makes this a simpler design challange.

Feeling inspired though which is good! Thanks for the questions… they are setting me thinking!

5

Thanks, I got into some further thinking, too (always good to get outside the own brain bubble).

This is probably much easier to achieve than the more complex settings you are hoping for.

You are probably right. The new shelf light I built spends most time in solid color, and I would be really happy if I just could adjust brightness with my wheel. Ah, in case you didnt stumble over it, this is my main project. I am open to put all this fancy stuff to the side and focus on a more simplified approach, even this would be a great benefit (kill your darlings, you know ;).

as I’m happy to use the phone app for those

This may be a difference in our approaches. I am not, meaning that I dont want to use my phone more, I want to use it less. Furthermore, as I am going for a retail product, my goal here is to get an MVP that works more or less phone-less (and even some of my tech oriented people surrounding me dont have a smart phone for reasons). As I have written I would be fine with a basic hardware UI with the option to make fancier stuff with phone/computer

like using short press… medium press… and long press

Have you rough ideas about the timings here? With some buttons I have problems always hitting the right timing even with only short and long (and double, thats also a tricky one). With adding medium long press should be…hmm…long. I also have thought about a “timeout” where the “cursor” is jumping back to the T1 menu. One utilization could be: 1. use wheel to dim 2a. when hitting button within X seconds, brightness is saved for the slot/preset and cursor jumps back one level 2b. when doing nothing for X secs, brightness is not saved and cursor jumps back one level. Just as an example how to save a command (e.g. long press) for some simple saving function. But these time based hidden actions should be used very carefully, as these hidden automated actions can easily confuse if not clear or used too often in too many ways. Especially if there is no index.

But it is interesting to think of ways to keep it very simple and intuitive (and more importantly looking simple!), for most users… but allow for more advanced usage for someone trained!

Definitely!
Another way of taking advantage of coded actions could be accessing the expert mode (my config mode) by a cheat code like sequence, or better (bcs. saver) the “hold knob down and move wheel” command (when actions are only executed on release this should be save).
One crucial advantage of such a mode is the complexity you can add without overwhelming a “normal” user, as you can anticipate a pro in front of the light who takes a look at the manual (manual, another important aspect, at least for me :wink: if she cant remember all the complex stuff. Or you take out you phone/computer and use it. I think thats also part of some user tests to see what ppl actually do and what not/rarely.

so that other advanced modes could be triggered with a magnet in the right place

I really like this idea. As I also aim for “playful control” having an element like a key-card could add some further benefit to the UI. Maybe for sth. like my mentioned “playing” use case a magnet can open such a mode. When having one not so symmetrical you can decode the direction into different modes (I have a fable for blackbox puzzles, but thats another story ^^).
I also thought of a NFC based element that can be used to add new effects (minor updates so to say) to the light. Or you have little cards, nicely illustrated with effects (or sth else), you can attach to the light within a light jukebox mode. Talking about that, having cards could be a proper and elegant solution for all this “effect chaos”, as you have the option to express all relevant info that a user must know.

Centre-push button on the knob to operate a momentary switch. Currently considering differences between those with detents and those without…

Hmm, my intuition tells me without detent you get problems with everything that is a list and not a (at least ordinary) fine-grained scale. Speaking of that, its important to know the hardware limitation, as I dont think you can change more than a few effects in realtime every second without having crushes or lags. Also here the “hold knob and move wheel” function could be of use (instead of acceleration maybe) to either take finer or rougher steps.

For my use case, the led feedback becomes simpler, as colours, or brightness, or warmth can be potrayed using a few leds, to give the idea of what will happen if you rotate left or right at any given point

When already having a knob, one of these 12 LED rings would be a great fit, as their cyclic structures would allow you to put all kind of indices into them (scales, previews, etc…) I will probably go for a dedicated display, but I dont take this into account atm as its a bonus, not a dependency.

Summing it up
We should aim for a normal user friendly “kiosk” mode and an advanced “config/expert” mode. My guts tells me that it also helps me to put all this mind boggling chaos to the side for a moment. Nevertheless I try to take everything into account, as the following other modes should be at least consistent with the basic UIs in some ways.
Though I am no coder myself (I can google error messages and copy code) I think if we get to a detailed and consistent model, we will find the needed help for implementation within these surroundings :slight_smile:
I think about a structure of a working document and open a new one (gdoc or so where you are invited to join me) to bring everything in a form and sketch new diagrams.

Greetz
Kelvin

6

I have been again thinking about that whole “how to use existing remotes with wled” thing. At home I have Milight products, as they do not only produce all kinds of different illuminations, but also have the best remotes on the market. Sadly they work via RF, but their proprietary 2,4GHz protocol was reverse engineered and upon that the milight-hub raised. Its basically an ESP8266 with an NRF24L01 (rf transceiver), so it cant be too complicated adding such a module to the wled ESP. The codes are hardwired within the remotes and easily can be sniffed by the hub, so linking these codes to specific wled commands cant be too challenging either.
With this approach a lot of different really good light remotes (including one with a jog wheel :wink: ) could be used, all IR based solutions I found are cheap plastic crap.
Maybe a way to go, too. :slight_smile:

7

While I think it is a fantastic idea and a welcome concept, I wonder how much the remote RF noise in the same frequency range as the wifi for an esp8266/esp32 would impact the smooth operation of WLED.

Worst case, there could be a milight-hub usermod that receives remote control codes and forwards them on to the correct WLED instance with the usermod’s json api - in case the noise really was a problem. In this way, the hub could use one remote for multiple WLED instances, figuring out which WLED instance the command is meant for, and forwarding to the right controller.

8

At least from my experience and what I have read there are no general noise problems even in high noise environments.
There might be different setups that could be utilized.

  1. Every WLED ESP has an attached NRF24L01, so a “hacked” milight hub would act as a NRF-NRF bridge
  2. The hub acts as a NRF-Wifi bridge, translating the NRF codes from the remote to something that WLED does understand (UDP? MQTT?GET?..?)
  3. A Usermod within the WLED firmware with an attached NRF directly translates the remote codes into something that WLED does understand
  4. More as a workaround, there still is the option to integrate the remotes into HASS via MQTT

But also the other direction is pretty interesting (controlling Milights via WLED).

  1. There is the option to control the bulbs via UDP servers, so I wonder if these bulbs could be integrated into the WLED ecosystem (But here my skills clearly end… but maybe an idea others can correlate with).
  2. As I used diyHue to integrate the milights into the hue ecosystem via the milight-hub, I wonder if the hue-emulator-bridge could be used to get the milight lights connected to WLED in any way.
9

I just read that the upcoming ESP32-H2 has a built-in IEEE 802.15.4 radio. With this we should be a step closer using the milight remotes with WLED lights…I think.
One day it will work… :smiley: