Computational imaging of light in flight

Abstract

Many computer vision tasks are hindered by image formation itself, a process that is governed by the so-called plenoptic integral. By averaging light falling into the lens over space, angle, wavelength and time, a great deal of information is irreversibly lost. The emerging idea of transient imaging operates on a time resolution fast enough to resolve non-stationary light distributions in real-world scenes. It enables the discrimination of light contributions by the optical path length from light source to receiver, a dimension unavailable in mainstream imaging to date. Until recently, such measurements used to require high-end optical equipment and could only be acquired under extremely restricted lab conditions. To address this challenge, we introduced a family of computational imaging techniques operating on standard time-of-flight image sensors, for the first time allowing the user to "film" light in flight in an affordable, practical and portable way. Just as impulse responses have proven a valuable tool in almost every branch of science and engineering, we expect light-in-flight analysis to impact a wide variety of applications in computer vision and beyond.

Recent advances in transient imaging: A computer graphics and vision perspective

“…Depending on the application domain, existing algorithms to simulate transient light transport trade off accuracy for speed. As a forward model for efficient reconstruction of the geometry of occluded objects, Hullin [2014] and Klein et al [2016] extended Smith et al's [2008] transient version of the radiosity method [Goral et al 1984] on the GPU. This method is limited to Lambertian surface reflections, and second-bounce interactions.…”

“…For the particular case of K = 2, this approach allows for a very fast reconstruction [Dorrington et al 2011;Godbaz et al 2012;Adam et al 2016;Naik et al 2015]. Wu et al [2012a;2014] noted that in streak images this Ksparse model is actually convolved by the sensor temporal point spread function (PSF) (Figure 5, left), which presents a Gaussian shape [Velten et al 2013]. Following this observation, the authors model Equation ( 7) as a sum of Gaussians instead:…”

“…In the presence of translucent objects or participating media, indirect illumination plays an important role. For the case of translucent, optically thick objects, Wu et al [2012a;2014] empirically observed that a single exponential decay accurately models subsurface scattering as:…”

Fast back-projection for non-line of sight reconstruction

“…Despite all these available hardware options, the reconstruction step is still a bottleneck, limiting the applicability of this technology in the wild. Different approaches have been proposed for reconstruction, either by solving a non-convex optimization on three-dimensional geometry [11,12] or a depth map [8], or by back-projecting the space-time captured image on a voxelized geometry representation [3,13]. In both cases, the large amount of data being processed, together with the complexity of the reconstruction algorithms, impose computation times in the order of several hours.…”

Fast back-projection for non-line of sight reconstruction

“…Different approaches have been proposed for reconstruction, either by solving a non-convex optimization on three-dimensional geometry [11,12] or a depth map [8], or by back-projecting the space-time captured image on a voxelized geometry representation [3,13]. In both cases, the large amount of data being processed, together with the complexity of the reconstruction algorithms, impose computation times in the order of several hours.…”