LIME: Live Intrinsic Material Estimation

Meka, Abhimitra; Maximov, Maxim; Zollhöfer, Michael; Chatterjee, A.; Seidel, Hans-Peter; Richardt, Christian; Theobalt, Christian

doi:10.1109/cvpr.2018.00661

Cited by 102 publications

(68 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LIME [26] Ours Object 1 0.45% 0.00037% Object 2 1.37% 0.14% Table 2: We compare the relative error between the estimated diffuse albedo for two objects. We outperform LIME even though our method is not restricted to the estimation of only a single material at a time.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Inverse Path Tracing for Joint Material and Lighting Estimation

Azinović

Li²,

Kaplanyan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

102

View full text Add to dashboard Cite

Inverse Path Tracing Roughness Emission Albedo Rendering Geometry & Target ViewsFigure 1: Our Inverse Path Tracing algorithm takes as input a 3D scene and up to several RGB images (left), and estimates material as well as the lighting parameters of the scene. The main contribution of our approach is the formulation of an end-to-end differentiable inverse Monte Carlo renderer which is utilized in a nested stochastic gradient descent optimization. AbstractModern computer vision algorithms have brought significant advancement to 3D geometry reconstruction. However, illumination and material reconstruction remain less studied, with current approaches assuming very simplified models for materials and illumination. We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation. We assume a coarse geometry scan, along with corresponding images and camera poses. The key contribution of this work is an accurate and simultaneous retrieval of light sources and physically based material properties (e.g., diffuse reflectance, specular reflectance, roughness, etc.) for the purpose of editing and re-rendering the scene under new conditions. To this end, we introduce a novel optimization method using a differentiable Monte Carlo renderer that computes derivatives with respect to the estimated unknown illumination and material properties. This enables joint optimization for physically correct light transport and material models using a tailored stochastic gradient descent.

show abstract

Section: Methodsmentioning

confidence: 99%

“…This problem has been viewed in the 2D image domain, resulting in a large body of work on intrinsic images or videos [1,27,26]. However, the problem is severely underconstrained on monocular RGB data due to lack of known geometry, and thus requires heavy regularization to jointly solve for lighting, material, and scene geometry.…”

Section: Introductionmentioning

confidence: 99%

Inverse Path Tracing for Joint Material and Lighting Estimation

Azinović

Li²,

Kaplanyan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

102

View full text Add to dashboard Cite

show abstract

“…Barron and Malik [2] recover geometry, reflectance and illumination from a single image of an arbitrary object by enforcing hand-crafted priors on each component. Recently, deep learning-based methods have been proposed to recover illumination and material properties (along with, in some cases, geometry) from a single RGB image of an object [8,17,22,16]. These methods do not easily scale to large-scale indoor scenes where the illumination, geometry, and reflectance properties are significantly more complex.…”

Section: Related Workmentioning

confidence: 99%

Deep Parametric Indoor Lighting Estimation

Gardner

Hold-Geoffroy

Sunkavalli

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

126

156

View full text Add to dashboard Cite

We present a method to estimate lighting from a single image of an indoor scene. Previous work has used an environment map representation that does not account for the localized nature of indoor lighting. Instead, we represent lighting as a set of discrete 3D lights with geometric and photometric parameters. We train a deep neural network to regress these parameters from a single image, on a dataset of environment maps annotated with depth. We propose a differentiable layer to convert these parameters to an environment map to compute our loss; this bypasses the challenge of establishing correspondences between estimated and ground truth lights. We demonstrate, via quantitative and qualitative evaluations, that our representation and training scheme lead to more accurate results compared to previous work, while allowing for more realistic 3D object compositing with spatially-varying lighting.

show abstract

“…the render loss (rloss) [19,23,38,15,28]; 2) the difference between the ground truth and the estimated parameters, i.e. the parameter loss (ploss) [20,24,28]. Methods that rely on the rloss are in general more precise as the training learns how much a variation of parameters affect the desired results.…”

Section: Isotropic Materialsmentioning

confidence: 99%

BRDF Estimation of Complex Materials with Nested Learning

Vidaurre

Casas

Garcés

et al. 2019

2019 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Gr o u n d T r u t h S y n t h e s i z e d S y n t h e s i z e d P r e d i c t i o n I n p u t I ma g e s I n p u t P h o t o sFigure 1: From just two input images (left) our method is capable of estimating the BRDF parameters (right, synthesized from novel viewpoints) of complex materials. AbstractThe estimation of the optical properties of a material from RGB-images is an important but extremely ill-posed problem in Computer Graphics. While recent works have successfully approached this problem even from just a single photograph, significant simplifications of the material model are assumed, limiting the usability of such methods. The detection of complex material properties such as anisotropy or Fresnel effect remains an unsolved challenge. We propose a novel method that predicts the model parameters of an artist-friendly, physically-based BRDF, from only two low-resolution shots of the material. Thanks to a novel combination of deep neural networks in a nested architecture, we are able to handle the ambiguities given by the nonorthogonality and non-convexity of the parameter space. To train the network, we generate a novel dataset of physicallybased synthetic images. We prove that our model can recover new properties like anisotropy, index of refraction and a second reflectance color, for materials that have tinted specular reflections or whose albedo changes at glancing angles.

show abstract

LIME: Live Intrinsic Material Estimation

Cited by 102 publications

References 32 publications

Inverse Path Tracing for Joint Material and Lighting Estimation

Inverse Path Tracing for Joint Material and Lighting Estimation

Deep Parametric Indoor Lighting Estimation

BRDF Estimation of Complex Materials with Nested Learning

Contact Info

Product

Resources

About