State of the Art on Neural Rendering

Abstract

Efficient rendering of photo‐realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo‐realistic images from hand‐crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo‐realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state‐of‐the‐art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photorealistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. Specifically, our emphasis is on the type of control, i.e., how the control is provided, which parts of the pipeline are learned, explicit vs. implicit control, generalization, and stochastic vs. deterministic synthesis. The second half of this state‐of‐the‐art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free‐viewpoint video, and the creation of photo‐realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

Roadmap on 3D integral imaging: sensing, processing, and display

“…Another emerging research area that provides a strong link between light field capture and display is neural scene representation and rendering (e.g., [104,105]). Instead of focusing too much on camera or display device development, these machine learningdriven methods take as input one of multiple views of a scene and distill them into a differentiable 3D scene representation, typically a neural network.…”

CryoGAN: A New Reconstruction Paradigm for Single-Particle Cryo-EMviaDeep Adversarial Learning

“…A GAN-based scheme that recovers the underlying distribution of the data from its noisy partial observations through a forward model was recently proposed in [27]. Finally, the reconstruction of a 3D structure (implicitly or explicitly) from its 2D viewpoints (and not projections) is an important problem in computer vision [53]. Many recent deep-learning algorithms have been used in this regard [54,55].…”

Light stage super-resolution

“…More advanced techniques were developed to produce a parametric model of the geometry and reflectance for even highly specular objects [Tunwattanapong et al 2013]. There are also works that focus on recovering a parametric model from a single image [Sengupta et al 2018], constructing a volumetric model for view synthesis [Lombardi et al 2018], or even a neural representation of a scene [Tewari et al 2020]. However, the complicated reflectance and geometry of human subjects is difficult to even parameterize analytically, let alone recover.…”