Predicting Novel Views Using Generative Adversarial Query Network

Abstract

The problem of predicting a novel view of the scene using an arbitrary number of observations is a challenging problem for computers as well as for humans. This paper introduces the Generative Adversarial Query Network (GAQN), a general learning framework for novel view synthesis that combines Generative Query Network (GQN) and Generative Adversarial Networks (GANs). The conventional GQN encodes input views into a latent representation that is used to generate a new view through a recurrent variational decoder. The proposed GAQN builds on this work by adding two novel aspects: First, we extend the current GQN architecture with an adversarial loss function for improving the visual quality and convergence speed. Second, we introduce a featurematching loss function for stabilizing the training procedure. The experiments demonstrate that GAQN is able to produce high-quality results and faster convergence compared to the conventional approach.Keywords: novel view synthesis · generative adversarial query network · mean feature matching loss

Sequential View Synthesis with Transformer

“…Recently, researchers have adopted deep learning techniques to overcome the inherent limitation of the conventional approaches. This paper focuses on a family of neural rendering methods [6,7,8,9,10] that infer the underlying 3D scene structure and faithfully produces the target view even at a distant query pose. These methods use an aggregate function to represent the entire 3D scene as a single implicit representation.…”

“…Recent neural rendering methods have introduced a generative model that understands the underlying 3D scene structure and faithfully produces the target view at the distant query pose [31,32,33,34]. Generative Query Network (GQN) [6] and its variant [7,8,9] are incorporating all input observation (images and poses) into a single implicit 3D scene representation to generate the target view. This aggregated representation contains all necessary information (e.g.…”