Understanding time-of-flight sensing

Introduction: ranging technologies

Humans and quite a few animals employ various senses for measuring their distance from other objects. Vision is the most commonly used. At night, the sense of touch can be used to feel the presence of other objects – and of course touch is a crucial sense for the visually impaired. The behavior of sound waves enables bats to use echo-location to navigate around obstacles, and many animals use sound in less sophisticated ways to estimate distance from objects.

Each sensory method for proximity measurement differs in its accuracy, range, and sophistication, and each has its own role to play in the natural world.

In the electronics world, a similar diversity has emerged in response to the requirements of different products and operating conditions. In a car, for instance, multiple sensor modes are in use:

• Radar, LiDAR and vision cameras perform high-speed object detection at long range
• Ultrasonic sensing offers the high precision at short range required in parking assistance systems
• Time-of-flight sensors detect the presence of occupants inside the cabin. By measuring the posture and position of the driver’s head, a time-of-flight system can support driver monitoring, an important safety function being introduced into new car designs
• Advanced capacitive sensors in the head unit’s touchscreen can detect the presence of the user’s hand at short range

Time-of-flight (ToF) sensing occupies its own distinctive space within this ecosystem of ranging methods. As a leading supplier of ranging sensors which enable the camera auto-focus function in smartphones, ams OSRAM has built up deep expertise in the implementation of this form of sensing. This white paper comes in response to growing interest in ToF sensing in the industrial and consumer markets: Customers which have not previously used a ToF sensor can draw on the knowledge which ams OSRAM brings from the smartphone market.

A study of the function of ToF sensing starts with a description of the two types of ToF sensor, and the ways in which they differ in their characteristics and uses.

The types of ToF sensor, and their mode of operation

As the sensor’s name suggests, a ToF sensor works by measuring the outward and return travel of repeated pulses of invisible infrared light which are reflected from objects within the sensor’s field of view.

There are two methods of measuring the travel of each pulse of light:

• In a direct time-of-flight (dToF) sensor, a light pulse is emitted, triggering a time-to-digital converter (TDC) to start. The TDC is used as a stopwatch and when the sensor module’s photodetector receives a reflected pulse, the time taken for the emitted pulse to return is stored. The time measured by the TDC is converted into a distance value, using the following formula:

• An indirect time-of-flight (iToF) sensor works by measuring the phase shift between the emitted and reflected pulses of light.

The dToF sensor type is small and consumes little power. A low-cost component of the bill-of-materials, it performs mm accurate distance measurements typically in the range of 0mm to several meters.

iToF sensors are capable of performing high-speed, high-resolution 3D imaging of objects at short and long range. An iToF sensor is typically larger than a dToF sensor, consumes more power, has a higher unit cost and due to the large number of pixels requires more complex processing. iToF sensors have only phase information for each pixel, iToF is not capable of differentiating different objects and may mix multiple objects into one pixel thus reporting a wrong distance.

Additionally changes of crosstalk are mixed with the phase information making iToF very sensitive to contamination of the cover glass. This is typically addressed by increasing the separation between the illuminator and receiver causing a much larger module than dToF would need.

To provide detailed ‘pictures’ of 3D space, a system might use an iToF sensor. In 3D applications, iToF sensing is rivalled by other technologies, including:

• Active stereo vision and structured light – commonly used to generate a depth map of a smartphone user’s face, enabling face authentication or face recognition to be implemented in sophisticated software

LiDAR systems generate a depth map which can be converted to a point cloud rendered as a 3D image at a range up to 150m enabling vehicles to navigate autonomously. The subject of this white paper, however, is dToF sensing. Advanced dToF sensors today can perform multi-object detection and distance measurement within their field of view. The newer multi-zone types can also detect an object’s direction of movement, enabling the host system to perform gesture sensing.

By comparing the distance from an object of different zones in the field of view, multi-zone dToF sensors can detect the angle of approach to an object, enabling navigation around objects as well as collision avoidance.

The sweet spot for dToF sensors’ application

True distance measurement – the ams OSRAM dToF sensors feature an ultra-high precision timing circuit and TDC to give typical distance measurement accuracy of
±1cm with a resolution of 1mm.

Low-power operation – the ams OSRAM dToF sensor modules contain a vertical- cavity surface-emitting laser (VCSEL) and closely coupled driver, which produces light pulses of picosecond duration. This short-pulse mode of operation consumes very little power – typical power consumption of a dToF sensors is measured in milliwatts. The low power of the IR pulses also means that the ams OSRAM dToF sensors are Class 1 certified for eye safety.

Robust high performance – the hardware implemented by ams OSRAM aggregates the TDC timings from multiple pulses into histogram bins. This histogram-based method enables the sensor to measure the distance of multiple objects with a high degree of confidence, even when the signal-to-noise ratio is reduced by bright ambient light, or by smudges or other contamination on the sensor’s cover window.

Small package size – dToF sensors are small, surface-mount modules (see Figure 2). The latest TMF882x multi-zone dToF sensors have a board footprint of just 4.6mm x 2.0mm and a package height of 1.4mm. The single zone TMF880X devices are even smaller measuring just 3.6mm x 2.2mm x 1.0mm.
This means that dToF sensors are particularly well suited to systems in which space and power are limited, and accurate object detection is required over a range of 5m or less.

Typical applications for dToF sensing include:

• Laser detect auto-focus (LDAF) – an essential function for both the front and world-facing cameras in smartphones
• Home service robots such as floor cleaners, in which the dToF sensor performs object avoidance and cliff detection. A multi-zone sensor can also navigate around objects
• User presence detection – in smart door locks and automatic door openers, a multi-zone dToF sensor detects a user as they approach a door. When a person is in range, the system can be triggered to open the door, or to start a high-power operation such as a face recognition system. A multi-zone dToF sensor can detect the direction of a person’s motion. So a person passing from one side of a door to the other will not trigger the system, whereas a person approaching a door front- on will trigger it.

User presence detection can also be used in laptop computers to switch off the screen when not in use. This provides security protection and saves battery power. If user presence detection is implemented with a multi-zone dToF sensor, the computer can also implement gesture sensing.

• Contactless switching – in automatic tap or toilet flush controls, for instance, the superior distance measurement capability of a low-power dToF sensor provides more reliable detection of the user’s hand than technologies typically in use today

• Gesture recognition – a dToF sensor may be used in displays, light switches, vending machines and elsewhere to provide contactless control of user functions.

Multi-zone dToF sensors from ams OSRAM can enable recognition of swipes, taps and circular gestures.

In fact, low cost, small size and low power consumption make dToF sensors attractive to a growing range of industrial and consumer product manufacturers. The implementation of dToF sensing, however, requires a basic understanding of the optical operation of the device.

This white paper provides a high-level introduction to the technology, and the main considerations which a design engineer has to take into account when designing a dToF sensor into their application.

Using a dToF sensor in the application: the role of the histogram

The principle of operation of a dToF sensor is simple. Implementing this principle in practice entails fairly complex hardware and software components, to ensure that the sensor maintains accurate performance even in adverse operating conditions.

As described above, the internal VCSEL emits multiple pulses of 940nm infrared light. Reflections from each pulse can be individually detected by the internal high- sensitivity SPAD (single photon avalanche diode) photodetector. The time measured by the TDC from each received pulse is plotted in a histogram (see Figure 3).

The histogram provides a robust method for the sensor module’s algorithm to determine the degree of confidence which applies to each object detection event. As Figure 3 shows, a histogram representation clearly distinguishes real-world objects in the field of view from noise generated by sources such as ambient light, optical crosstalk (which will be explained below), and contamination on the cover glass.

The TMF882X sensor processes the histogram and provides its calculated measurement result as a digital signal over an I2C interface. This digital signal consists of:

• A distance value for the closest and the second closest object in the field of view, in 1mm increments
• An 8-bit degree of confidence value for each object, where a value of 255 indicates extremely high confidence. The degree of confidence reflects the signal-to-noise ratio (SNR) of each histogram peak.

This concept of the degree of confidence underpins an important configuration option which ams OSRAM dToF sensors afford to the user. The histogram measures an aggregate number of ‘counts’, or reflected pulses: the higher the number of counts, the higher the SNR. The operation of the dToF sensor’s VCSEL emitter can be configured to optimize either for power consumption or SNR. At a lower pulse frequency, the sensor uses less power, and at a higher pulse frequency, the sensor’s SPAD will produce more counts. By default, the TMF882x multi-zone sensors emit at a frequency of 17.7 MHz: the end user can change the number of VCSEL pulses as needed, trading power consumption for improved performance.

The number of pulses (termed the ‘iteration setting’ in the datasheet) is configurable by the user on the fly. A common configuration in battery-powered applications is to set the sensor in a lower power mode, operating at a low pulse frequency to save energy. When an uncertain measurement result is obtained, or in the presence of bright ambient light (which reduces the signal to noise ratio), the pulse frequency can be increased with a view to providing measurement results which have the required degree of confidence.

Using a dToF sensor in the application: optical configuration

In a single-zone dToF sensor, the closest object detected within the fixed field of view will produce peaks in the histogram.

The multi-zone dToF sensors from ams OSRAM give the user the flexibility to adjust the field of view to the requirements of the application. Figure 4 shows simple examples of the ways in which the field of view might benefit from a particular configuration of the zones.

The TMF8820 provides a 3×3 multi-zone array, the TMF8821 is a 4×4 zone sensor, and the TMF8828 offers a 64-pixel 8×8 array.

The sensors provide a variety of options for configuring the array’s field of view. The sensors are supplied with a set of pre-configured ‘SPAD masks’ – those for 3×3 operation are shown in Figure 5.

In addition, customers can design their own custom SPAD maps. The TMF8820 supports up to nine zones and the TMF8821 supports up to 18 zones in a time- multiplexed mode.

Because of the sensor’s short focal length of just 400µm – a consequence of the small size of the module’s package – the area of each zone becomes extremely large at a long distance. This has a marked effect on the applications which are most commonly implemented, such as gesture sensing. The sensor will only reliably detect a hand moving across multiple zones when the hand is close to the sensor – typically less than 2m.

Using a dToF sensor in the application: optical design considerations

As an optical device, a dToF sensor requires careful consideration of the properties of the cover glass which protects the sensor from damage.

The considerations which apply to the cover window are:

• Shape
• Angle relative to the sensor
• Thickness and air gap
• IR transmissivity
• Contamination
• Calibration

While the cover window’s primary purpose is physical protection of the sensor, it also has an optical effect on the beams of light emitted and received by the sensor. The first principle determining the design of the cover window should be to distort or attenuate the light path to the least possible extent to maintain the highest possible range and accuracy.

Shape and angle

A curved cover window will bend the light away from the intended target. Refraction will also occur if the cover glass deviates in any way from a perpendicular angle relative to the beam from the sensor’s VCSEL emitter.

In a product which has a circular enclosure such as a robotic floor cleaner, there can be a temptation to use a curved cover window which can be mounted flush with the circular enclosure. This temptation should be resisted, as the curvature of the glass substantially distorts the optical pathway: this creates great difficulties in calculating distance measurements, and in determining the shape and position of the field of view.

For the same reason, the cover window should be mounted perpendicular to the VCSEL’s angle of emission.

Thickness

A proportion of the light emitted from the sensor’s VCSEL never reaches the target, but instead is reflected from the inner or outer surface of the cover window (see Figure 6). Some of this reflected light will reach the SPAD array, generating crosstalk which raises the ambient noise level, decreasing the signal to noise ratio of the system.

Below a certain threshold level of crosstalk, the sensor’s histogram-based algorithm cancels out this crosstalk. But it is important to follow good design practices which minimize crosstalk.

The two most important are:

• Minimize the thickness of the cover glass. In thin glass, reflections from the outer surface bounce more times internally before reaching the SPAD array. With each bounce, the beam’s optical power is attenuated. In thick cover glass, the beam bounces fewer times. So less crosstalk is generated by thin cover glass.
• Minimize the air gap between the top of the sensor and the bottom of the cover glass. The larger the air gap, the more crosstalk is generated. For accurate long- distance ranging up to 5m, the air gap should be kept to a maximum of 1mm.

Please refer to the ams OSRAM Optical Design Guides for further details on optimal configurations.

Transparency and coatings

The ams OSRAM dToF sensors emit and detect narrow-bandwidth infrared light at 940nm. For reasons of industrial design, many products require the use of inked glass. It is important that the ink should be transparent to IR light.

The material chosen for the cover window should itself be highly transparent to light at 940nm. The recommendation of ams OSRAM is that the material chosen for the cover should be at least 85% transparent. Tempered glass and polycarbonate are both suitable materials for use in dToF applications.

Other factors which affect optical transmission through the cover, such as surface roughness, embedded particles or trapped air, should be kept to a minimum. The total effect of any such imperfections should be optical attenuation of less than 6%.

If any coatings are applied to the cover for aesthetic or industrial design reasons, for instance to color the cover to match the product’s enclosure, uncoated windows or exclusion areas should be reserved to allow for full transmission of light from the emitter and to the SPAD array.

Contamination

Contamination of the cover window increases the amount of crosstalk. Contamination can be caused by dust resting on the surface of the window, and by smudges – grease and dirt transferred from a user’s fingers.

As stated above, the ams OSRAM dToF sensors can maintain range and accuracy with a certain amount of crosstalk. A cover window which produces high levels of crosstalk when clean will be highly sensitive to contamination by smudges.

The design should allow for the additional effect of crosstalk created by contamination, and evaluate its effect on the application’s requirement for range and accuracy.

Calibration

Every product design makes its own, unique optical assembly formed by the combination of cover window material, thickness, angle and shape, and the air gap between the sensor and cover window.

This assembly must be factory-calibrated using guidelines provided by ams OSRAM in the sensor’s datasheet. The host application should upload the calibration data to the sensor each it starts up. In addition, calibration data are unique to each SPAD map, so a calibration data set should be generated for each SPAD map used in the application, and the appropriate set uploaded to the sensor when the SPAD map changes during operation.

The calibration settings are sensitive to tolerances in the production of the optical assembly. Consistency in factors such as the air gap and cover window thickness should be maintained to a high level of precision.

ams OSRAM dToF devices offer simple calibration, using the final optical stack-up, no target within 40cm and minimal ambient light. For maximum performance, the system crosstalk should be within the guidelines highlighted within the Optical Design Guide.

How to start developing a dToF sensor-based ranging system

This white paper has outlined the basic principles of operation of a direct time-of- flight sensor, and the important optical design issues to consider. For first-time users of a dToF sensor, however, the biggest advances in understanding will come with active evaluation of a dToF sensor, and its integration into a prototype design.
The design engineer is substantially supported in this effort by the broad range of products, tools and resources available from ams OSRAM.

dToF products

ams OSRAM supplies a range of fully integrated dToF sensors which offer either single- or multi-zone ranging capability.

Single-zone dToF sensors for industrial, consumer and medical applications are the TMF8801 and TMF8805, which produce accurate distance measurements over a range between 2cm and 2.5m.

The TMF8701 single-zone sensor is primarily intended for short range sensing applications including in front-facing smartphone cameras. It combines proximity sensing capability with auto-focus ranging between 10cm and 60cm.

All three single-zone dToF sensors are housed in a 2.2mm x 3.6mm x 1.0mm surface- mount package.

Multi-zone dToF sensors are the 3×3 multi-zone TMF8820, the 4×4 multi-zone TMF8821, and the 8×8 multi-zone TMF8828. These products produce distance measurements over a range between 1cm and 5m. The field of view is adjustable up to 63°.

The ams OSRAM multi-zone dToF sensors are housed in a 12-lead OLGA package which has a footprint of 2.0mm x 4.6mm, and which is just 1.4mm high. Its volume is less than half that of the leading competitor.

dToF evaluation tools

ams OSRAM supplies dedicated evaluation kits which provide a ready-made environment for evaluating different configurations of the sensor settings, and different optical assemblies.

The TMF882x-EVM is a Raspberry Pi-based evaluation module to support development of applications within a Linux® operating environment. It is backed by a full-featured GUI front end which displays at a glance:

• All measured distances
• All measured histograms
• Histograms per zone in matrix format)
• Measurement parameters
• Information about the operation of the module

An example of the measured distances display is shown in Figure 7.

The GUI also provides a 2D heat map and 3D rendering of objects’ distance measurements, as shown in Figure 8. The higher the confidence in the measurement, the more solid the color block appears. The GUI’s data-logging feature also enables the user to extract raw histogram data for offline analysis.

ams OSRAM also supplies a dedicated evaluation tool for microcontroller users. The TMF882X-SHIELD is a plug-in board in the Arduino™ UNO form factor. It provides a flexible development platform for quick evaluation of the TMF8820, TMF8821 and TMF8828 multi-zone sensors (see Figure 9).

The boards consist of a 20mm x 12mm sensor break-away board carrying the dToF sensor. This is attached to the Arduino main board. The dToF target board can easily be incorporated into the user’s prototype system.

The TMF882x Shield board is shipped with four cover window samples of different thicknesses, and four air gap spacers of different depths. This gives the user a convenient way to test the performance of various optical assemblies in their application.

To complement the TMF882X-SHIELD, ams OSRAM also provides Arduino- compatible libraries for the TMF882X devices which enable customers to quickly begin development, and eventually port code to their own hardware platform. With these libraries, the developer can quickly change device configurations: this helps them to understand the effects of changing parameters such as the number of iterations, the measurement period, or the SPAD maps.

Python drivers are available which can be used with an I2C to USB interface board and the TMF882X_SHIELD.

A software development kit (SDK) for the NXP LPC55S69 microcontroller is also available.

dToF sensor development resources

A comprehensive set of documentation is available online to support development with any ams OSRAM dToF sensor. This includes:

• TMF8801 Product Datasheet
• TMF8805 Product Datasheet
• TMF882X Product Datasheet
• TMF882x optical design guide, which provides information on building the optical stack, crosstalk limits, configuring the field of illumination and field of view, and the use of an optical isolation boot.

• Host driver communication application report, supporting device set-up and operation
• TMF882X-EVM Evaluation Kit
• TMF882X-EVM user guide
• TMF882X-SHIELD Evaluation kit
• TMF882X-SHIELD Quick Start Guide

The online hub for information about the ams OSRAM dToF sensors and short-range proximity sensors is at ams OSRAM’s website.

Biography:

David Smith is a Senior Product Marketing Manager for 1D Time of Flight products within the Advanced Optical Solutions group at ams OSRAM. He has over 20 years’ experience in applications, sales, business development and product marketing within the semiconductor industry.