Fast Online 3D Reconstruction of Dynamic Scenes From Individual Single-Photon Detection Events

Abstract

In this paper, we present an algorithm for online 3D reconstruction of dynamic scenes using individual times of arrival (ToA) of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon Lidar is the integration time required to build ToA histograms and reconstruct reliable 3D profiles in the presence of non-negligible ambient illumination. This long integration time also prevents the analysis of rapid dynamic scenes using existing techniques. We propose a new method which does not rely on the construction of ToA histograms but allows, for the first time, individual detection events to be processed online, in a parallel manner in different pixels, while accounting for the intrinsic spatiotemporal structure of dynamic scenes. Adopting a Bayesian approach, a Bayesian model is constructed to capture the dynamics of the 3D profile and an approximate inference scheme based on assumed density filtering is proposed, yielding a fast and robust reconstruction algorithm able to process efficiently thousands to millions of frames, as usually recorded using single-photon detectors. The performance of the proposed method, able to process hundreds of frames per second, is assessed using a series of experiments conducted with static and dynamic 3D scenes and the results obtained pave the way to a new family of real-time 3D reconstruction solutions.

Expectation-Maximization Based Approach to 3D Reconstruction From Single-Waveform Multispectral Lidar Data

“…Thus, future work should consist of further accelerating the inference process. An interesting route for improvement could be online reconstruction, as in [58].…”

Robust 3D Reconstruction of Dynamic Scenes From Single-Photon Lidar Using Beta-Divergences

“…As mentioned in the introduction, our online estimation procedure consists of leveraging the temporal correlation between successive frames by incorporating the posterior distribution of the depth profile at time (n − 1) in the inference problem at time n. As described in [30], estimating the posterior mean and variance of d p,n presents a significant advantage beyond simply providing summary statistics about the current range profile. It allows the derivation of tractable adaptive estimation procedures.…”

“…Whilst this approach is simple, it does not allow for rapid changes as might occur when the imaging system or the scene moves orthogonally to the direction of observation. To alleviate issues associated with such changes while keeping the estimation strategy tractable, we define as in [30], for each pixel, a local neighborhood V p of M neighbors (including the current pixel) and define the following prior model…”

“…Thus, there is a clear need for scalable and reliable methods able to adaptively process the increasing amount of single-photon data recorded by new single-photon avalanche diode (SPAD) detector arrays [28], [29], offering a growing number of pixels. In [30], we proposed an online reconstruction method, relying on individual photon-detection events, e.g., binary frames, which was used to reconstruct sequentially a series of depth images using at most one photon per pixel and frame. Although this sequential approach leverages correlations between successive time frames, the method does not allow the analysis of histograms (i.e., reconstruction after several illumination periods), as the estimation is performed after each illumination period.…”

“…This integration period is userdefined and we assume that the scene is quasi-static during that period. In contrast to [30], the sequential reconstruction by the proposed algorithm is performed after an arbitrary number of periods and thus does not make any assumptions regarding the number of detection events to be used in the reconstruction of each point cloud. The most basic and fastest method to estimate the distance of an object from a SPL histogram is via log-matched filtering but this method fails if the background level due to ambient illumination and light scattering is too high.…”