Tracking objects outside the line of sight using 2D intensity images

Klein, Jonathan; Peters, Christoph; Martin, J.P.; Laurenzis, Martin; Hullin, Matthias B.

doi:10.1038/srep32491

Cited by 140 publications

(94 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiments, we use Lambertian and metal BRDFs, but other reflectance functions can be used as well. This approximation can be seen as an extension of the one found in [Klein et al 2016]. We further add two important features to increase physical realism and generate a smooth transient image.…”

Section: Rendering (Synthesis)mentioning

confidence: 95%

“…A smaller but more diverse group of work relies on the use of forward models to arrive at a scene hypothesis that best agrees with the measured data. Here, reported approaches range from combinatorial labeling schemes [Kirmani et al 2009] via frequency-domain inverse filtering (if the capture geometry is sufficiently constrained) [O'Toole et al 2018a] to variational methods using simple linearized light transport tensors [Heide et al 2014;Naik et al 2011] and simplistic models based on radiative transfer [Klein et al 2016;Pediredla et al 2017] that are (in principle) capable of expressing opacity effects like shadowing and occlusion, and physically plausible shading and that are closest to our proposed method. In concurrent work, Heide et al [2017] added such extra factors as additional weights into their least-squares data term, achieving non-line-of-sight reconstructions of significantly improved robustness.…”

Section: Analysis Of Transient Light Transport and Looking Around Cormentioning

confidence: 99%

See 1 more Smart Citation

Non-line-of-sight Reconstruction Using Efficient Transient Rendering

Iseringhausen

Hullin

2020

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

Object Object(a) (b) (c) Fig. 1. (a) The challenge of looking around the corner deals with the recovery of information about objects beyond the direct line of sight. In this illustration of a setting proposed by Velten et al. [2012], an unknown object is located in front of a wall, but additional obstacles occlude the object from any optical devices like light sources or cameras. Our only source of information are therefore indirect reflections off other surfaces (here, a planar "wall"). A point on the wall that is illuminated by an ultrashort laser pulse turns into an omnidirectional source of indirect light ("laser spot"). After scattering off the unknown object, some of that light arrives back at the wall, where it forms an optical "echo" or space-time response (shown are 2D slices) that can be picked up by a suitable camera. Locations on the wall can be interpreted as omnidirectional detector pixels that receive different mixtures of backscattered light contributions at different times. We assume that neither camera nor laser can directly illuminate or observe the object, leaving us with the indirect optical space-time response as the only source of information. Note that for the sake of clarity, laser source, camera, and occluder are not shown here. The complete setup is illustrated in Figure 2. (b) We propose a novel transient renderer to simulate such indirectly scattered light transport efficiently enough for use as a forward model in inverse problems. In this artistic visualization, light contributions removed by the shadow test are marked in red, and the net intensity in blue. Together with an optimization algorithm, the renderer can be used to reconstruct the geometry of objects outside the line of sight. (c) Left to right: ground-truth object geometry; reconstruction using a state-of-the-art method (ellipsoidal backprojection); reconstruction using the technique presented in this paper. Top row: BunnyGI dataset; bottom row: Mannequin1Laser dataset. Our method relies on highly efficient and near-physical forward simulation, and it exemplifies the use of computer graphics as a technical tool to solve inverse problems in other fields.Being able to see beyond the direct line of sight is an intriguing prospective and could benefit a wide variety of important applications. Recent work has demonstrated that time-resolved measurements of indirect diffuse light contain valuable information for reconstructing shape and reflectance properties of objects located around a corner. In this paper, we introduce a novel reconstruction scheme that, by design, produces solutions that are consistent with state-of-the-art physically-based rendering. Our method combines an efficient forward model (a custom renderer for time-resolved three-bounce indirect light transport) with an optimization framework to reconstruct object geometry in an analysis-by-synthesis sense. We evaluate our algorithm on a variety of synthetic and experimental input data, and show that it gracefully handles uncooperative scenes with high levels of no...

show abstract

Section: Rendering (Synthesis)mentioning

confidence: 95%

Section: Analysis Of Transient Light Transport and Looking Around Cormentioning

confidence: 99%

Non-line-of-sight Reconstruction Using Efficient Transient Rendering

Iseringhausen

Hullin

2020

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Still, several works have been proposed to deal with light transport in a time‐resolved manner. In particular, most previous work on transient rendering has focused on simulating surfaces transport: Klein et al [KPM*16]. extended Smiths' transient radiosity [SSD08] for second bounce diffuse illumination, while other work has used more general methods based on transient extensions of Monte Carlo (bidirectional) path tracing [Jar12, JMM*14, PBSC14, JA18] and photon mapping [MNJK13, OHX*14].…”

Section: Related Workmentioning

confidence: 99%

Progressive Transient Photon Beams

Marco

Guillén

Jarosz

et al. 2019

Computer Graphics Forum

View full text Add to dashboard Cite

In this work, we introduce a novel algorithm for transient rendering in participating media. Our method is consistent, robust and is able to generate animations of time‐resolved light transport featuring complex caustic light paths in media. We base our method on the observation that the spatial continuity provides an increased coverage of the temporal domain, and generalize photon beams to transient‐state. We extend stead‐state photon beam radiance estimates to include the temporal domain. Then, we develop a progressive variant of our approach which provably converges to the correct solution using finite memory by averaging independent realizations of the estimates with progressively reduced kernel bandwidths. We derive the optimal convergence rates accounting for space and time kernels, and demonstrate our method against previous consistent transient rendering methods for participating media.

show abstract

“…Different approaches have been proposed for reconstruction, either by solving a non-convex optimization on three-dimensional geometry [11,12] or a depth map [8], or by back-projecting the space-time captured image on a voxelized geometry representation [3,13]. In both cases, the large amount of data being processed, together with the complexity of the reconstruction algorithms, impose computation times in the order of several hours.…”

Section: Introductionmentioning

confidence: 99%

“…Further work explored alternative hardware setups, including correlation-based time-of-flight cameras [8], single photon avalanche diodes (SPAD) [9], laser-gated sensors [10], or even common DSLR cameras [11]. These setups are cheaper and more portable, although at the cost of sacrificing time resolution.…”

Section: Introductionmentioning

confidence: 99%

Fast back-projection for non-line of sight reconstruction

2017

View full text Add to dashboard Cite

Recent works have demonstrated non-line of sight (NLOS) reconstruction by using the time-resolved signal from multiply scattered light. These works combine ultrafast imaging systems with computation, which back-projects the recorded space-time signal to build a probabilistic map of the hidden geometry. Unfortunately, this computation is slow, becoming a bottleneck as the imaging technology improves. In this work, we propose a new back-projection technique for NLOS reconstruction, which is up to a thousand times faster than previous work, with almost no quality loss. We base on the observation that the hidden geometry probability map can be built as the intersection of the three-bounce space-time manifolds defined by the light illuminating the hidden geometry and the visible point receiving the scattered light from such hidden geometry. This allows us to pose the reconstruction of the hidden geometry as the voxelization of these space-time manifolds, which has lower theoretic complexity and is easily implementable in the GPU. We demonstrate the efficiency and quality of our technique compared against previous methods in both captured and synthetic data.

show abstract

Tracking objects outside the line of sight using 2D intensity images

Cited by 140 publications

References 27 publications

Non-line-of-sight Reconstruction Using Efficient Transient Rendering

Non-line-of-sight Reconstruction Using Efficient Transient Rendering

Progressive Transient Photon Beams

Fast back-projection for non-line of sight reconstruction

Contact Info

Product

Resources

About