Why The Best Hand Tracking Camera Is Infrared

Posted; August 13, 2021

The reasons for needing our world-leading hand tracking are diverse: to create a truly immersive VR experience, to accurately recreate a medical scenario in VR for training, or to reliably and quickly trigger button pushes on the screen of an interactive kiosk.

Our hand tracking cameras need to be robust, but versatile enough to work in a wide variety of environments and lighting conditions.

Ultraleap's hand tracking software takes raw data on the position, orientation, and velocity of your hands and turns it into a real-time digital model.

While in theory the raw data for hand tracking can come from anywhere – an RGB camera, GPS, laser arrays, LiDAR – in practice we've found that using infrared cameras and LEDs provides the most consistent and reliable data in a variety of lighting conditions.

In nature, creatures that use infrared vision tend to be able to move between highly variable environments. Using infrared helps our hand tracking cameras be similarly versatile.

Here’s why:

Whether it’s a Leap Motion Controller 2, a Stereo IR 170, or an integrated headset built using the Qualcomm XR2 reference design, they all use two infrared cameras combined with infrared LEDs inside the camera module.

Eyes are uniquely adapted to detect a vast range of brightness. However, a camera can only "see" about a 100 times difference in brightness. This is a major limitation. In real-life environments light levels are hugely variable and unpredictable.

This is a problem we solve by providing a consistent light source from within the hand tracking cameras themselves. Instead of relying on available light, our camera modules are equipped with infrared LEDs that light up your hands.

Providing the light means we can control the light. We can compensate for variation in environmental light levels and provide a predictable light source. This makes our hand tracking system very robust.

Using LEDs that emit visible light to illuminate your hands would also work from a purely technical perspective. However, it wouldn't make for a good user experience. Especially in dimly lit environments, visibly illuminating hands would be distracting.

In contexts such as theme park dark rides this would impact the experience. Using infrared means we can light hands effectively – but invisibly. Only the hand tracking cameras can see it.

Like visible light, infrared light is a type of electromagnetic radiation. Electromagnetic radiation is produced when atoms absorb and then release energy.

There is a spectrum of frequencies, from high frequency X-rays and ultraviolet radiation, to lower frequency radio-waves. Infrared wavelengths range from around 1,000 micrometers to 760 nanometers. This is lower than visible light, which is why humans aren’t able to see infrared.

Sensing and detection is one of the most impactful uses of infrared. Everything emits infrared radiation, and this can be detected by thermal imaging sensors like those used in night vision goggles and security cameras.

It’s a tool also used by NASA when they want to be able to see things that would otherwise be obscured. NASA’s Cassini spacecraft uses infrared cameras to filter out the haze surrounding Saturn. Using infrared can provide a focus on what’s important by looking at things in a way not possible with typical cameras.

Unlike thermal imaging sensors, Ultraleap’s hand tracking cameras don’t rely on the infrared naturally emitted by your hands. Instead, they use infrared LEDs to light up your hands. The infrared is what the cameras "see".

We use infrared to show us what matters (and filter out what doesn’t).

Ultraleap’s hand tracking cameras don’t track all infrared light. They "see" wavelengths of around 850 nanometres – a narrow band within the infrared spectrum.

An infrared filter – a tiny square of material that sits just above the cameras – filters out all wavelengths that are not between 825 to 875 nanometers in length. The remaining wavelengths of light are in the precise area of the infrared spectrum covered by our LEDs.

This small detail helps to filter out excess light in the environment that might otherwise overexpose the camera image, and provides the sharpest images possible. Focusing on this narrow spectrum creates the optimal image for hand tracking – clear hands against a dark background.

Providing infrared light to brightly illuminate the hands in this way helps us control the image in brighter environments. It also means reliable and accurate hand tracking in dim environments.

Intelligent input from the software can also modify how the hand tracking camera works. For example, if the software detects the room is very dark, it sends back a command to dim the LEDs.

Likewise, if your hands are moving slowly the software may slow down the camera’s frame-rate. This responsive behaviour both improves tracking accuracy and uses less power.

A final trick is that the hand tracking cameras don’t continuously illuminate your hands with infrared. Our patented technique works like a camera flash, with the cameras sending pulses of power to the LEDs at incredibly short intervals.

The LEDs pulse in perfect synchronicity with the camera frame-rate, producing flashes of infrared light at exactly the moment the camera shutter opens. With this pulse trick we use much less power.

Inside the cameras, the light reflected from your hands is focused onto the image sensor by the lens.

The image sensor, although small, contains hundreds of thousands of light-capturing wells. These are called photosites. Photosites convert photons to electrons and the electric current is converted into a digital light value called a pixel.

The data is input into our tracking engine as pixels. Following some serious machine learning magic, our software outputs, in real time, 27 points in three-dimensional space. These are the 27 skeletal points that our hand tracking cameras and software track.

Applications (such as Unity or Unreal) then build on top of this data to create 3D interactive hand models. It can also be used for other purposes.

TouchFree, for example, takes the same data and uses it to create a hand-controlled cursor. Our haptics products use the data to tell the haptic system where to output a sensation.

No matter what industry you're working in, to create effective applications you need to design around the minimum consistent performance of a system. This means that it's not merely hand tracking but reliable hand tracking that enables transformational and scalable experiences.

Our hand tracking cameras and software are the most robust and flexible solution in the world. We'd love to work with you to explore how they could transform your applications.