Real-time illumination estimation from faces for coherent rendering

Knorr, Sebastian; Kurz, Daniel

doi:10.1109/ismar.2014.6948416

Cited by 25 publications

(39 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The approach of Knorr and Kurz [15] estimates incident light from human faces using machine learning from a variety of faces under different illumination conditions. The method applies a face tracker and matches distinctive observation points of the detected face to the training database to estimate the real-world light.…”

Section: Related Workmentioning

confidence: 99%

“…We empirically found a space of 2 15 illumination variations sufficient, which is created by setting the ambient light C 0 = 1 and quantizing the higher-order SH coefficients C 1 to C 15 with two different values in the range of -1 to 1. Note, selecting only two different values for each coefficient keeps the size of the database small enough for training the CNNs in an acceptable amount of time.…”

Section: Synthesis Of Training Imagesmentioning

confidence: 99%

“…For each image set, we systematically vary the environmental lighting l j . The resulting database for a camera pose P i consists of 2 15 images T i, j .…”

Section: Synthesis Of Training Imagesmentioning

confidence: 99%

See 2 more Smart Citations

Learning Lightprobes for Mixed Reality Illumination

Mandl

Mohr

et al. 2017

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

View full text Add to dashboard Cite

Figure 1: Coherent Illumination. The real scene consists of an action figure of The Hulk and a toy car. To demonstrate the result of our system, we place a 3D scan next to the real action figure and display it using Mixed Reality: (left) The 3D reconstruction is rendered without real world light estimation. (right) Our system estimates the current lighting from a single input image. The estimated lighting is used to illuminate the 3D reconstruction. Note that we only register the real world lighting and do not consider any camera effects such as exposure or blur. ABSTRACTThis paper presents the first photometric registration pipeline for Mixed Reality based on high quality illumination estimation by convolutional neural network (CNN) methods. For easy adaptation and deployment of the system, we train the CNN using purely synthetic images and apply them to real image data. To keep the pipeline accurate and efficient, we propose to fuse the light estimation results from multiple CNN instances, and we show an approach for caching estimates over time. For optimal performance, we furthermore explore multiple strategies for the CNN training. Experimental results show that the proposed method yields highly accurate estimates for photo-realistic augmentations.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Synthesis Of Training Imagesmentioning

confidence: 99%

See 1 more Smart Citation

Learning Lightprobes for Mixed Reality Illumination

Mandl

Mohr

et al. 2017

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

View full text Add to dashboard Cite

show abstract

“…However these approaches cannot handle inserting a specific model, but rather proposes a set of images of objects in a specific category that roughly match illumination conditions in the desired photograph. Recently, Knorr and Kurz [KK14] proposed a framework for estimating the real-world lighting conditions based on the captured appearance of a human face. The method is based on learning a face-appearance model from an offline dataset of faces under known illumination.…”

Section: Estimated Lighting Conditionsmentioning

confidence: 99%

Photorealistic rendering of mixed reality scenes

Kronander

Banterle

Gardner

et al. 2015

Computer Graphics Forum

View full text Add to dashboard Cite

Photo-realistic rendering of virtual objects into real scenes is one of the most important research problems in computer graphics. Methods for capture and rendering of mixed reality scenes are driven by a large number of applications, ranging from augmented reality to visual effects and product visualization. Recent developments in computer graphics, computer vision, and imaging technology have enabled a wide range of new mixed reality techniques including methods for advanced image based lighting, capturing spatially varying lighting conditions, and algorithms for seamlessly rendering virtual objects directly into photographs without explicit measurements of the scene lighting. This report gives an overview of the state-of-the-art in this field, and presents a categorization and comparison of current methods. Our in-depth survey provides a tool for understanding the advantages and disadvantages of each method, and gives an overview of which technique is best suited to a specific problem.

show abstract

“…Lee and Woo [11] propose a method which reflects the illumination of a scene into the rendering of virtual objects by referring to the reflection on a fiducial marker. Knorr and Kurz [12] estimate a scene illumination from the reflection of light on human faces, and realize coherent AR visualization with the estimated illumination. Fischer et al [13] incorporate the sensor noise and the motion blur of a camera when rendering virtual objects by blurring and adding synthetic noise on an output image.…”

Section: Visual Consistency In Ar Applicationsmentioning

confidence: 99%

Semi-Parametric Color Reproduction Method for Optical See-Through Head-Mounted Displays

Itoh¹,

Dzitsiuk²,

Amano

et al. 2015

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

The fundamental issues in Augmented Reality (AR) are on how to naturally mediate the reality with virtual content as seen by users. In AR applications with Optical See-Through Head-Mounted Displays (OST-HMD), the issues often raise the problem of rendering color on the OST-HMD consistently to input colors. However, due to various display constraints and eye properties, it is still a challenging task to indistinguishably reproduce the colors on OST-HMDs. An approach to solve this problem is to pre-process the input color so that a user perceives the output color on the display to be the same as the input. We propose a color calibration method for OST-HMDs. We start from modeling the physical optics in the rendering and perception process between the HMD and the eye. We treat the color distortion as a semi-parametric model which separates the non-linear color distortion and the linear color shift. We demonstrate that calibrated images regain their original appearance on two OST-HMD setups with both synthetic and real datasets. Furthermore, we analyze the limitations of the proposed method and remaining problems of the color reproduction in OST-HMDs. We then discuss how to realize more practical color reproduction methods for future HMD-eye system.

show abstract

Real-time illumination estimation from faces for coherent rendering

Cited by 25 publications

References 28 publications

Learning Lightprobes for Mixed Reality Illumination

Learning Lightprobes for Mixed Reality Illumination

Photorealistic rendering of mixed reality scenes

Semi-Parametric Color Reproduction Method for Optical See-Through Head-Mounted Displays

Contact Info

Product

Resources

About