Fast Non-line-of-sight Imaging with Two-step Deep Remapping

Abstract

Conventional imaging only records the photons directly sent from the object to the detector, while non-line-of-sight (NLOS) imaging takes the indirect light into account. To explore the NLOS surroundings, most NLOS solutions employ a transient scanning process, followed by a back-projection based algorithm to reconstruct the NLOS scenes. However, the transient detection requires sophisticated apparatus, with long scanning time and low robustness to ambient environment, and the reconstruction algorithms typically cost tens of minutes with high demand on memory and computational resources. Here we propose a new NLOS solution to address the above defects, with innovations on both detection equipment and reconstruction algorithm. We apply inexpensive commercial Lidar for detection, with much higher scanning speed and better compatibility to real-world imaging tasks. Our reconstruction framework is deep learning based, consisting of a variational autoencoder and a compression neural network. The generative feature and the two-step reconstruction strategy of the framework guarantee high fidelity of NLOS imaging. The overall detection and reconstruction process allows for real-time responses, with state-of-the-art reconstruction performance. We have experimentally tested the proposed solution on both a synthetic dataset and real objects, and further demonstrated our method to be applicable for full-color NLOS imaging.

Recent Advances on Non-Line-of-Sight Imaging: Conventional Physical Models, Deep Learning, and New Scenes

“…5) LiDAR: Although most existing works have used separate light sources and detectors, recent work [47] exploited a commercial LiDAR, which integrated light source and detector, to complete NLOS imaging. For the imaging area on the relay surface that is close to LiDAR, most of the collected information is the direct reflection from the relay surface.…”

“…Deep-inverse correlography [14] Deep matrix factorization [67] Physics-based DeepRL framework [40] 3D reconstruction Learned feature embeddings [13,47] Conventional methods Modulated light source + TOF camera [3,46] Active light source + Interferometer [30,44,45] Laser + Conventional camera [15,48,[49][50][51] ToF-based [1,3,11,24,27] Wave-based [5] Phasor field [4,10,45] Backprojection [1,2,26] Inverse methods [3,11,21,22] Surface normal algorithms [24,30,33,62] Wave-based / Phasor field [4,5,10,45] Conventional camera + Ambient light [6,67,[73][74][75][76][77] Conventional camera + Incoherent source [8,68] Interferometer + Incoherent...…”

“…Pulsed laser SPAD Time of fight Detection/ Tracking/ Identification [40] Pulsed laser SPAD Time of fight Pose estimation [41], [42] Pulsed laser SPAD array Time of fight 3D reconstruction [43] Pulsed laser SPAD array Time of fight Detection/ Tracking/ Identification [30], [44], [45] Pulsed laser Interferometer Coherence 3D reconstruction [3], [46] Modulated laser ToF camera Time of fight 3D reconstruction [47] Integrated LiDAR Depth + intensity 3D reconstruction [15], [48] Continuous…”

“…For traditional physical models, using commercial LiDAR as the equipment is still a very challenging problem. However, the recent work of Zhu et al showed that by performing deep learning to complete feature extraction, LiDAR can be used to complete high-quality and robust 3D NLOS reconstruction [47].…”

Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging

“…Shen et al [6] introduced neural transient field (NeTF) for the implicit modeling of hidden objects. Zhu et al [26] developed a deep generative model for NLOS imaging using inexpensive commercial LiDAR.…”