Gesture Control inside a car

Design guide for gesture control in automotive HMI

Ultraleap’s Experience Design and Research team is building the standards for common gestures, interactions, and haptic sensations. Here, Matt Corrall, Design Director, shares our thinking around best practices for gesture control in automotive HMI.

The automotive HMI of the future is going to use a host of new technologies to respond more intelligently to a driver’s needs, increasing safety and letting you control your car in a way that’s natural and easy. It will likely be powered by facial and gesture recognition, voice control, visual displays, eye tracking, and haptic technology.

Ultraleap’s world-leading hand tracking powers reliable, customizable gesture control in automotive HMI – allowing drivers to control their car’s infotainment system with simple hand gestures.

Groupe PSA automotive HMI powered by gesture control
Automotive HMI powered by gesture control with mid-air haptics enables drivers to control infotainment intuitively and without taking their eyes off the road. With thanks to Groupe PSA.

Adding haptic feedback to gesture control using our “virtual touch” haptic technology gives tactile confirmation that an action has been completed, even though there’s no button or touchscreen present. Think of it as feeling a “force field” against your palm.

In our rigorous testing, it’s proven to result in drivers making fewer glances away from the road. Drivers also strongly prefer it over either gesture control alone, or touchscreens.

However, gesture control is new enough that there are no established standards yet for how to do things well – unlike, say, designing for a touchscreen. Our team is building those standards for common gestures, interactions, and haptic sensations.

Here, we share our current thinking on best-practice gesture frameworks for automotive HMI.

Design guidelines for automotive HMI gestures

Gesture Design Guide cover

Automotive HMI

In Ultraleap’s Experience Design and Research team, we study human behaviour and use evidence-led design practice to determine the gestures, interactions, and haptic sensations that work best. We work with customers integrating our technology into a wide range of products, using what we’ve learned to help them design intuitive, touchless experiences.

Gesture and interactions aren’t the same in every case. Our gesture choices will be affected by the context and job the user is doing, as well as whether they’re using a 2D screen or a VR headset.

Are they a driver operating an infotainment system without taking their eyes off the road, or an architect reviewing a detailed 3D building model in VR? For each case, we start with the person and find the interactions that best fit their needs.

Recently, Groupe PSA’s new luxury concept car the DS Aero Sport Lounge featured Ultraleap’s hand tracking and mid-air haptic technology built into the futuristic car’s central armrest. Powered by Ultraleap, the driver (or passengers) can control a range of systems with natural hand gestures – from entertainment to navigation.

Ultraleap haptic technology added to DS Automobiles car interior ui
Groupe PSA’s new luxury concept car the DS Aero Sport Lounge features Ultraleap’s hand tracking and mid-air haptic technology built into the futuristic car’s central armrest.

Designing infotainment with gesture control and mid-air haptics

The gesture interactions in the DS Aero Sport Lounge concept car were based on insights from a previous Ultraleap infotainment system design project. These were validated by our  designers in the XDR team through rigorous user testing and iterative design.  

The scope of the design project was determined by the study of market-leading infotainment systems and qualitative user research with drivers from a wide range of backgrounds. This gave us a sense of how automotive HMI is commonly used and let us determine the most important functions.

Based on this, we designed a prototype gesture control infotainment system that allowed users to:

  • Control climate through temperature and fan speed
  • Control music through track selection and volume
  • Handle navigation /maps, including setting a new route
  • Answer/decline phone calls
  • Switch between functions with a menu system

Our team designed and built a working prototype of the automotive HMI, conducting tests with 42 different users using a purpose-built driving simulator very similar to those used in the automotive industry. We then iterated upon the design and repeated our tests, learning which combination of hand gestures, haptic sensations, and on-screen UI components worked best together.

User with Ultraleap automotive haptic tracking demo
Prototype of Ultraleap's automotive HMI infotainment system, controlled by gesture with mid-air haptics

Designing gesture control for reliability

With gesture control, we need to consider the reliability with which the technology can recognise a gesture and interpret the user’s intention. Unlike pressing buttons on a touchscreen, hand gestures are analogue – my hand goes through a range of motions and postures as it forms a gesture and then releases it.

Hand shapes and sizes, the way people gesture and even ambient light will vary hugely, making it a technical challenge to get it right every time.

The hand tracking camera must also be mounted where it can get a good view of the user’s hands – often on the ceiling or dashboard. This avoids occlusion (i.e. when it’s not clear what gesture I am making because part or all of my hand is blocked from the camera’s view).

During user testing, our team assesses hand gestures and associated haptics and UI elements in terms of their usability, their comfort, and their reliability.

The following are our current recommendations for using hand gestures in automotive HMI, each with suggested functions, pros and cons. They’re from an in-depth study by Gareth Young, our Senior UX Designer.

Picking the right gestures depends on many factors and must be considered as part of a wider design system that includes haptics, UI components and audio feedback, but our guidelines are the best place to start.

Matt Corrall is Design Director at Ultraleap

Design guidelines for automotive HMI gestures

Terminology

Usability

Refers to how easy it is to learn, understand and remember the required form and movement of the gesture in order to perform it correctly. A gesture with high usability is considered easy to use relative to other gestures.

Reliability

Refers to the likeliness of the hand tracking device correctly interpreting the gesture. A gesture with high reliability can be expected to be recognised the vast majority of the time.

Comfort

Refers to how physically difficult a gesture is to perform, and how much discomfort is experienced whilst performing the gesture. A gesture with high comfort can be performed easily with minimal effort.

Hand Tap

Hand to tap gesture interaction by Ultraleap

Using a flat hand, the driver “bounces” their whole hand down and up with a fast motion.

Examples of actions it could be used for in car HMIs:

  • Yes
  • Confirm
  • Accept
  • Play / Pause (Toggle)
  • On / Off (Toggle)

Usability: Medium   |   Reliability: Medium   |   Comfort: High

Fingers Tap

Fingers tap gesture interaction by Ultraleap

Keeping their whole arm still, the driver moves their fingers (or a single finger) down and up with a fast motion. Requires less physical effort than the Hand Tap gesture.

Examples of actions it could be used for in car HMIs:

  • Select
  • Yes
  • Confirm
  • Accept
  • Play / Pause
  • Toggle On / Off

Usability: Medium   |   Reliability: Medium   |   Comfort: Medium

Pinch (with arm movement)

Pinch with arm movement gesture interaction by Ultraleap

The driver brings their index fingertip and thumb together whilst keeping their remaining fingers open (exposing their palm to feel mid-air haptic feedback).

It is the same as the universal sign for “OK”, and can be either be held for adjusting values/objects, or performed quickly to serve as a mid-air confirmation action.

They can also move their hand up whilst performing the pinch gesture to increase a value (e.g. volume), and down to decrease the value, before opening their hand to confirm the choice.

Finger poses for automotive interactions

This same gesture also works whilst moving in other directions, going left/right and forwards/backwards to control sliders, menus, and objects.

Examples of actions it could be used for in car HMIs:

  • Adjusting values (volume, temperature, etc.)
  • Navigating a 2D or 3D menu
  • Moving forwards/backwards through music or video
  • Picking up and dropping objects in AR/VR
  • Selecting a button that has been highlighted using a hand-controlled cursor

Usability: Medium   |   Reliability: High   |   Comfort: High

Grab-Release

Grab release gesture interaction by Ultraleap

The driver clenches their hand – making a fist - before opening it again. This is performed in one fluid motion.

Examples of actions it could be used for in car HMIs:

  • No
  • Cancel
  • Reject
  • Close
  • Reset
  • Off

Usability: High   |   Reliability: Low   |   Comfort: High

Finger Poses

Finger poses for automotive interactions
Finger pose gesture interaction Ultraleap

The driver simply shows a certain number of fingers to the hand tracking device. This can be used – for example – as a shortcut to quickly move between infotainment modes like music, navigation etc.

Examples of actions it could be used for in car HMIs:

  • Selecting a particular feature or mode
  • Choosing the desired amount of a particular item

Usability: High   |   Reliability: Medium   |   Comfort: Low

Subtle Swipe

Subtle swipe gesture interaction by Ultraleap

The driver “flicks” their hand from left to right, then back again in one single motion. Only the forearm needs to move, rather than the whole arm.

Examples of actions it could be used for in car HMIs:

  • Skipping a music track
  • Toggling between feature modes

Usability: High   |   Reliability: Low   |   Comfort: Medium

Hand Twist

Hand twist gesture interaction by Ultraleap

Starting with their hand open and palm facing down, the driver turns their hand over then back again in one fluid motion (hand rotates outwards rather than inwards).

Examples of actions it could be used for in car HMIs:

  • Back to previous screen
  • Cancel
  • Go to beginning of currently playing track / video
  • Play previous track / video

Usability: High   |   Reliability: High   |   Comfort: Medium

Summary

Ultraleap Gesture Control library table

Download Design Guidelines

By signing up, you are agreeing to our privacy policy.