Gated2Depth: Real-Time Dense Lidar From Gated Images

Abstract

We present an imaging framework which converts three images from a gated camera into high-resolution depth maps with depth accuracy comparable to pulsed lidar measurements. Existing scanning lidar systems achieve low spatial resolution at large ranges due to mechanicallylimited angular sampling rates, restricting scene understanding tasks to close-range clusters with dense sampling. Moreover, today's pulsed lidar scanners suffer from high cost, power consumption, large form-factors, and they fail in the presence of strong backscatter. We depart from point scanning and demonstrate that it is possible to turn a low-cost CMOS gated imager into a dense depth camera with at least 80 m range -by learning depth from three gated images. The proposed architecture exploits semantic context across gated slices, and is trained on a synthetic discriminator loss without the need of dense depth labels. The proposed replacement for scanning lidar systems is real-time, handles back-scatter and provides dense depth at long ranges. We validate our approach in simulation and on real-world data acquired over 4,000 km driving in northern Europe. Data and code are available at

Millimeter Wave MIMO-Based Depth Maps for Wireless Virtual and Augmented Reality

“…Previous depth map construction approaches focused on leveraging: (i) monocular images using RGB cameras [2], (ii) passive/active stereo images using either RGB-D depth cameras [3], [4] or infrared (IR) stereo cameras [5], [6], and (iii) gated images using active gated imaging cameras [7], [8]. In [2], a monocular depth estimation approach capable of capturing the object boundaries is proposed.…”

ISETAuto: Detecting Vehicles With Depth and Radiance Information

“…LiDAR systems typically sweep a laser light through the scene and measure the time-of-flight using a small array of avalanche photodetectors, and more recently single-photon avalanche detectors (SPADs). Gated imaging sensor systems also determine depth based on time-of-flight [16]. These systems emit a very brief near-infrared illumination pulse or a periodic temporal pattern and then precisely control (gate) a sequence of very brief electronic exposure times.…”

Ellipsoidal path connections for time-gated rendering

“…The recent decade saw a proliferation of advanced driver assistance systems (ADAS) to increase automobile safety, with fixed time-gated cameras of gate widths as narrow as 200 ps used to avoid collision and improve night-time navigation [David et al 2006;Grauer and Sonn 2015]). Coupled with machine learning algorithms, time-gated sensors are also used as alternatives to lidar for dense depth sensing [Gruber et al 2019]. Gated laser ranging and short-wave-infrared sensors are used in military applications, to survey objects at specific distances [las 2019;Baker et al 2004].…”

“…In turn, the availability of rendering algorithms greatly enhances the ability of the sensor designers to optimize their designs, resulting in novel sensors of superior performance. In particular, as state-of-the-art machine learning tools are increasingly used for both designing the sensors [Marco et al 2017a] and processing their outputs [Gruber et al 2019]. Efficient rendering tools are necessary to generate the large, diverse, and realistic datasets needed for training.…”