Time of Flight (ToF) Sensors on Smartphones Explained

by Mayank

In recent years, smartphone innovations have slowed down a bit. As a result, smartphone manufacturers have encouraged their development teams to create some groundbreaking features for their flagship devices that will set them apart from the competition. One of these features is the Time of Flight (ToF) sensor on smartphone cameras.

Despite a wide variety of applications, you can find very little knowledge on ToF technology on the Internet. This is because only a few experts in this field can provide a clear and concise explanation of this technology. In this article, I’m going to explain the ToF sensors, their operation and their wide range of applications in the simplest way possible.

Contents:

  1. What are ToF Sensors?
  2. How does a Time-of-Flight Camera Work?
  3. The Goods & Bads of ToF Camera Sensors
  4. Where are ToF Sensors used?
  5. Conclusion

What are ToF Sensors?

Camera technology has been in existence for a long time. As you may well know, when we click a picture, the camera shutter opens for some time to allow the light to reach the image sensor. Then, using some electronic circuitry and software magic, the light that entered is transformed into an image we see on our smartphone or DSLR.

This technology is fine for basic applications, but when you move to applications such as Computer Vision, Robotics, Augmented Reality, etc., this technology simply doesn’t help. Why? Because these applications require depth sensing to precisely determine the size and shape of the object. That’s where the Time of Flight sensor technology comes in.

Now, ToF sensors may sound new to you, but the technology has been in existence for more than three decades. Remember those old war movies? In those movies underwater submarines use SONAR to locate enemy ships. What happens is that a sound is emitted from the submarine in various directions.When this sound hits another vessel, it’s reflected back. This reflected sound is detected by the submarine and the time it took for the sound to travel the entire trip (from the submarine to the enemy vessel and back) is used to determine the exact location and distance of the enemy vessel.

The SONAR image of a Soviet Navy Minesweeper

The Time of Flight technology is similar to SONAR, but instead of using sound data, you are using light data.

The basic principle of ToF camera involves sending out a light signal and measuring the properties of the returned signal from a target. The measured data is used to determine the Time of Flight and the distance between the target and the ToF sensor is obtained by multiplying the Time of Flight and velocity (Distance= Time of Flight ×Velocity).

Note: Here velocity is the speed of light (3×108 m/s).

How does a Time-of-Flight Camera System Work?

Note: This section contains some physics terms that you may find difficult to understand.

A typical ToF-based camera sensor consists of a light source or an illumination unit. A laser or LED is used as a modulated source of light. This light hits the target and is reflected in various directions. A dimmer version of the original light reaches the image sensor. This reflected light has been shifted in phase by some degrees from the original light source. The image sensor has a pixel array in which each pixel is capable of storing phase information from the light as a charge. The difference in phase between the light source and the reflected light is used to measure the distance between the sensor and the target. Also, a lens is used as a focusing mechanism to focus the reflected light correctly on the image sensor. It is important that the light source and the reflected light are precisely synchronized in order to obtain the accurate distance.

Working of a ToF sensor

The widely used ToF sensors are CMOS-based ToF sensors. This is a single chip solution consisting of a CMOS pixel structure that can extract phase information from the light. In addition, each pixel is capable of filtering the light to reduce noise.

Architecture of a CMOS-based ToF sensor

The Goods & Bads of ToF Camera Sensors

Goods:

CMOS-based ToF technology is power efficient, cheap, has higher frame rates (up to 160 fps) and better resolution (320×240 pixels being the most common) compared to other types of depth sensors (such as triangulation systems that required two optical systems). The small size of the module makes it possible to implement it inside smartphone cameras. Also there is a need for less processing in CMOS-based ToF cameras, which helps reduce computing power requirements.

Bads:

  • Interference from background light: Background sunlight can interfere with the artificial light from the ToF sensor. The pixels must provide a good dynamic range to suppress background noise.
  • Multiple Reflections: As explained above, the reflected light from the target reaches the image sensor from several directions at different times. It is therefore possible that a portion of the reflected light could have been reflected from two different surfaces. This can cause a change in the time it takes for the light to reach the image sensor. This will cause the distance measured to be smaller than the actual distance.
  • Lower Resolutions: Although resolution is not a major problem for ToF sensors, it limits the applications of these sensors. This may well be the reason that Apple did not use the ToF sensor for its Face ID technology.

Read more: Learn more about the Face ID technology here.

Where are ToF Sensors used?

I’ve been talking about measuring distances since the beginning of this article. However, measuring distances is only a small part of the wide range of applications of ToF cameras. Here I’m going to list some applications where ToF cameras are used. But before that here’s a quick note:

Note: The list includes some smartphone applications, but you must know that at this point only a few flagship phones have a ToF sensor in their camera module.

Here’s a list of some smartphones that have a ToF camera sensor built-in to make it more clearer:

  • Samsung Galaxy S10 5G
  • Samsung Galaxy Note 10+
  • Samsung Galaxy S20+
  • Samsung Galaxy S20 Ultra 5G
  • LG G8 Thin Q
  • OPPO RX17 Pro
  • Honor View 20
  • Huawei P30 Pro
  • Huawei Mate 30 Pro
  • Huawei P40 Pro
  • Oppo Reno

The above list was provided by Root Nation.

Applications of ToF Sensors:

  • Measure Distance: You would have predicted this one. The best example of this would be the Measure application developed by Google. The application can measure the height and width of any object you want. Just open the application and point your camera to the object, and you’ll get a live measurement of the size of the object.
  • Background blur when taking photos: The 3D depth map generated by the ToF sensor can be used to separate the background and the foreground when taking a picture. This will help the image processing algorithm to get better information on the edges of an object to get the perfect background blur for awesome macro shots. In addition, if the ToF sensor is implemented to the front camera sensor, this will help in taking better portrait selfies. However, it is important to know that good software optimization is also needed.
The ToF Sensor enables better macro shots
  • Automotive Applications: Time-of-Flight cameras are also used for assistance and safety features in cars, such as pre-crash detection and pedestrian safety. Autonomous driving standards also require a vehicle to be well aware of the surroundings, and data needs to be updated frequently, requiring an instant 3D mapping of the surroundings that can be achieved through ToF cameras.
  • Gestures: Using ToF cameras, it is easy to track human movements in real-time, which opens the door to a wide range of applications, such as smartphone and television gesture controls. Using gesture controls on TVs, you can switch channels, increase the volume, turn on or off the TV. Smartphones such as the LG G8 ThinQ incorporate a ToF sensor on the front panel that allows for a wide range of gesture controls.
The LG G8 ThinQ features various kinds of gestures using 3D mapping by a ToF sensor
  • Computer Vision: Computer Vision involves the acquisition, processing and analysis of data from real-world environments through digital photos and videos. This data can then be used by the system to automate tasks based on user preferences. It also involves the object and motion tracking used by the second generation of the Xbox Kinect sensor, which has a ToF sensor built-in.
  • Robotics: The most famous application of ToF cameras is in the field of robotics where robots use it to quickly map their surroundings, avoid obstacles and facilitate human interactions.
Robots interact to objects using 3D mapping of their environment

Conclusion

If you ask me, ToF is one of the most innovative technologies of the decade. The use of this technology is on the rise and will have a huge impact on consumer electronics in some time. Especially in the case of smartphones, features such as better portrait modes, night visions and AR are more than good reasons to buy a smartphone equipped with a ToF sensor. So, fingers crossed, let’s see how manufacturers will use this technology in their future products.

Related Posts

Leave a Comment