Why The Best Hand Tracking Camera Is Infrared
Posted; August 13, 2021
Our hand tracking cameras need to be robust, but versatile enough to work in a wide variety of environments and lighting conditions.
Ultraleap's hand tracking software takes raw data on the position, orientation, and velocity of your hands and turns it into a real-time digital model.
While in theory the raw data for hand tracking can come from anywhere – an RGB camera, GPS, laser arrays, LiDAR – in practice we've found that using infrared cameras and LEDs provides the most consistent and reliable data in a variety of lighting conditions.
In nature, creatures that use infrared vision tend to be able to move between highly variable environments. Using infrared helps our hand tracking cameras be similarly versatile.
Seeing things differently – Why infrared hand tracking cameras?
Eyes are uniquely adapted to detect a vast range of brightness. However, a camera can only "see" about a 100 times difference in brightness. This is a major limitation. In real-life environments light levels are hugely variable and unpredictable.
This is a problem we solve by providing a consistent light source from within the hand tracking cameras themselves. Instead of relying on available light, our camera modules are equipped with infrared LEDs that light up your hands.
We can control the light
Providing the light means we can control the light. We can compensate for variation in environmental light levels and provide a predictable light source. This makes our hand tracking system very robust.
Using LEDs that emit visible light to illuminate your hands would also work from a purely technical perspective. However, it wouldn't make for a good user experience. Especially in dimly lit environments, visibly illuminating hands would be distracting.
In contexts such as theme park dark rides this would impact the experience. Using infrared means we can light hands effectively – but invisibly. Only the hand tracking cameras can see it.
What exactly is infrared?
Like visible light, infrared light is a type of electromagnetic radiation. Electromagnetic radiation is produced when atoms absorb and then release energy.
There is a spectrum of frequencies, from high frequency X-rays and ultraviolet radiation, to lower frequency radio-waves. Infrared wavelengths range from around 1,000 micrometers to 760 nanometers. This is lower than visible light, which is why humans aren’t able to see infrared.
Sensing and detection is one of the most impactful uses of infrared. Everything emits infrared radiation, and this can be detected by thermal imaging sensors like those used in night vision goggles and security cameras.
It’s a tool also used by NASA when they want to be able to see things that would otherwise be obscured. NASA’s Cassini spacecraft uses infrared cameras to filter out the haze surrounding Saturn. Using infrared can provide a focus on what’s important by looking at things in a way not possible with typical cameras.
Unlike thermal imaging sensors, Ultraleap’s hand tracking cameras don’t rely on the infrared naturally emitted by your hands. Instead, they use infrared LEDs to light up your hands. The infrared is what the cameras "see".
We use infrared to show us what matters (and filter out what doesn’t).
Our hand tracking cameras only use a narrow band of infrared
Ultraleap’s hand tracking cameras don’t track all infrared light. They "see" wavelengths of around 850 nanometres – a narrow band within the infrared spectrum.
An infrared filter – a tiny square of material that sits just above the cameras – filters out all wavelengths that are not between 825 to 875 nanometers in length. The remaining wavelengths of light are in the precise area of the infrared spectrum covered by our LEDs.
This small detail helps to filter out excess light in the environment that might otherwise overexpose the camera image, and provides the sharpest images possible. Focusing on this narrow spectrum creates the optimal image for hand tracking – clear hands against a dark background.
Providing infrared light to brightly illuminate the hands in this way helps us control the image in brighter environments. It also means reliable and accurate hand tracking in dim environments.
Intelligent input from the software can also modify how the hand tracking camera works. For example, if the software detects the room is very dark, it sends back a command to dim the LEDs.
Likewise, if your hands are moving slowly the software may slow down the camera’s frame-rate. This responsive behaviour both improves tracking accuracy and uses less power.
Patented pulsing power
A final trick is that the hand tracking cameras don’t continuously illuminate your hands with infrared. Our patented technique works like a camera flash, with the cameras sending pulses of power to the LEDs at incredibly short intervals.
The LEDs pulse in perfect synchronicity with the camera frame-rate, producing flashes of infrared light at exactly the moment the camera shutter opens. With this pulse trick we use much less power.
From pixels to 3D digital hands
Inside the cameras, the light reflected from your hands is focused onto the image sensor by the lens.
The image sensor, although small, contains hundreds of thousands of light-capturing wells. These are called photosites. Photosites convert photons to electrons and the electric current is converted into a digital light value called a pixel.
The data is input into our tracking engine as pixels. Following some serious machine learning magic, our software outputs, in real time, 27 points in three-dimensional space. These are the 27 skeletal points that our hand tracking cameras and software track.
No matter what industry you're working in, to create effective applications you need to design around the minimum consistent performance of a system. This means that it's not merely hand tracking but reliable hand tracking that enables transformational and scalable experiences.
Our hand tracking cameras and software are the most robust and flexible solution in the world. We'd love to work with you to explore how they could transform your applications.