Editable free-viewpoint video using a layered neural representation

Abstract

Generating free-viewpoint videos is critical for immersive VR/AR experience, but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper, we propose the first approach for editable free-viewpoint video generation for large-scale view-dependent dynamic scenes using only 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity, including the environment itself, is formulated into a spatiotemporal coherent neural layered radiance representation called ST-NeRF. Such a layered representation supports manipulations of the dynamic scene while still supporting a wide free viewing experience. In our ST-NeRF, we represent the dynamic entity/layer as a continuous function, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further

NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

“…[ Zhang et al 2021b] further supports certain spatial and temporal editing functions based on dynamic layered neural representations. [Peng et al 2021; use the parametric human model as prior to learn a dynamic radiance field for human body using sparse views as inputs.…”

“…[ Pumarola et al 2020;Zhang et al 2021b] learn deformation field from multi-view videos and optimize a radiance field in canonical space, their approach supports a larger viewing range and better rendering quality compared to previous approaches.…”

“…To process NeuVV, the brute-force approach would be to directly train a NeRF using the factorization, as in previous video-based NeRF [Pumarola et al 2020;Zhang et al 2021a]. Its downside is that NeRF is not readily supportive for real-time rendering, critical for video viewing.…”

“…The approach suffers from several artifacts: it inherently incurs high memory consumption, slow training, and cross-frame inconsistency/flicking. Alternative approaches such as ST-NeRF [Zhang et al 2021b], D-NeRF [Pumarola et al 2020], NeuralBody [Peng et al 2021] and HumanNeRF ] conduct spatial-temporal warping to map individual frames to a common canonical space so that they only need to train a single NeRF. The quality relies heavily on the accuracy of the estimated warping field; when deformation is large or the performer contains too few or too many textures, they tend to produce strong visual artifacts.…”

“…Most notably, the Neural Radiance Field (NeRF) [Mildenhall et al 2020] replaces the traditional notion of geometry and appearance with a single neural network where any new camera views can be realistically rendered by querying respective rays from the camera via neural inference. Despite its effectiveness, NeRF and its extensions have been largely focused on the static object, with a few exceptions [Kasten et al 2021;Lu et al 2020;Zhang et al 2021a] to directly tackle dynamic scenes. Further, existing solutions are still a few orders of magnitudes slower than real-time to support immersive volumography.…”