Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

Abstract

Fig. 1. We propose the use of cheap, small off-the-shelf distance sensors (far left) for a variety of computational imaging and vision tasks, and demonstrate and evaluate their capabilities in emerging sensing applications like (from left to right) material classification, non-line-of-sight tracking, and depth imaging.Time-correlated imaging is an emerging sensing modality that has been shown to enable promising application scenarios, including lidar ranging, fluorescence lifetime imaging, and even non-line-of-sight sensing. A leading technology for obtaining time-correlated light measurements are singlephoton avalanche diodes (SPADs), which are extremely sensitive and capable of temporal resolution on the order of tens of picoseconds. However, the rare and expensive optical setups used by researchers have so far prohibited these novel sensing techniques from entering the mass market. Fortunately, SPADs also exist in a radically cheaper and more power-efficient version that has been widely deployed as proximity sensors in mobile devices for almost a decade. These commodity SPAD sensors can be obtained at a mere few cents per detector pixel. However, their inferior data quality and severe technical drawbacks compared to their high-end counterparts necessitate the use of additional optics and suitable processing algorithms. In this paper, we adopt an existing evaluation platform for commodity SPAD sensors, and modify it to unlock time-of-flight (ToF) histogramming and hence computational imaging. Based on this platform, we develop and demonstrate a family of hardware/software systems that, for the first time, implement applications that had so far been limited to significantly more advanced, higher-priced setups: direct ToF depth imaging, non-line-of-sight object tracking, and material classification.

Plastic Classification Using Optical Parameter Features Measured with the TMF8801 Direct Time-of-Flight Depth Sensor

“…Our work complements Ref. [ 17 ] by presenting results captured by a direct ToF sensor with improved timing resolution that features 128 histogram bins at a timing resolution of 200 ps per bin. We evaluate classification over a range of sample-to-sensor distances so that a classification system that is robust to distance is created.…”

“…Since these sensors are designed for smartphones, the system is low-power and the illumination is eye-safe. Callenberg et al used a commercially available direct ToF sensor (ST VL53L1X [ 16 ]) to classify materials [ 17 ]. The sensor is placed in direct contact, and the return time histogram data are used as features to train a convolutional neural network that identifies foam, paper, skin, towel, and wax with an accuracy greater than 90% [ 17 ].…”

“…Callenberg et al used a commercially available direct ToF sensor (ST VL53L1X [ 16 ]) to classify materials [ 17 ]. The sensor is placed in direct contact, and the return time histogram data are used as features to train a convolutional neural network that identifies foam, paper, skin, towel, and wax with an accuracy greater than 90% [ 17 ]. By placing the sensor in direct contact with the material and varying the region of interest (ROI) of the detector, this system is sensitive to angular and spatial variations of the returned optical transient.…”

Neural Volumetric Reconstruction for Coherent Synthetic Aperture Sonar

“…Many works use optical transient imaging for measuring scenes in depth by leveraging continuous wave time-of-flight devices [Heide et al 2013;Kadambi et al 2013Kadambi et al , 2016 or impulse-based time-offlight single photon avalanche diode (SPADs) [Callenberg et al 2021;O'Toole et al 2017]. In particular, transient imaging is useful for nonline-of-sight (NLOS) reconstruction [Ahn et al 2019;Arellano et al 2017;Lindell et al 2019b;Liu et al 2019;Pediredla et al 2019;Velten et al 2012].…”

3D Scene Inference from Transient Histograms

“…Further, this two-step method can only work in the presence of a collocated RGB camera. Callenberg et al [4] use a 2-axis galvo-mirror system with a low-cost proximity sensor to scan the scene. This leads to an impractical acquisition time of more than 30 minutes even for a relatively low 128 × 128 resolution scan.…”