No abstract

Figure 1: We capture a video around a target subject (the Egyptian bust) and we re-enact the target's face in novel viewpoints. Our re-enactment is driven by an expression sequence of a source subject captured using a custom app running on an iPhone.

Recently neural volumetric representations such as neural reflectance fields have been widely applied to faithfully reproduce the appearance of real-world objects and scenes under novel viewpoints and lighting conditions. However, it remains challenging and time-consuming to render such representations under complex lighting such as environment maps, which requires individual ray marching towards each single light to calculate the transmittance at every sampled point. In this paper, we propose a novel method based on precomputed Neural Transmittance Functions to accelerate the rendering of neural reflectance fields. Our neural transmittance functions enable us to efficiently query the transmittance at an arbitrary point in space along an arbitrary ray without tedious ray marching, which effectively reduces the time-complexity of the rendering. We propose a novel formulation for the neural transmittance function, and train it jointly with the neural reflectance fields on images captured under collocated camera and light, while enforcing monotonicity. Results on real and synthetic scenes demonstrate almost two order of magnitude speedup for renderings under environment maps with minimal accuracy loss.

No abstract

Figure 1: Approach overview: Given (a) a dataset of multi-view images and segmentation masks of a category-specific object, along with (b) a single template mesh with UV coordinates, our method trains a network (c) exploiting information from multiple views using reprojection cycles and learn an instance-specific mesh using deformations. (d) At test time, our model predicts an instance-specific mesh and a surface mapping from a single image.