InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images

Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask

“…Using layered depth images, 3DPhoto [63] generates free-viewpoint videos from a single image, and it can be trained on the in-the-wild dataset. Following [36,38,58] that directly use official pretrained models for evaluation, we also use the official pretrained models for our evaluation. We combine 3DPhoto and SPADE by synthesizing a color image from the semantic mask using SPADE, then applying 3DPhoto to generate a novel view image.…”

“…AdaMPI [20] proposes a warp-back training strategy to train MPI on the COCO [5] datasets. Li et al [36] propose a cycle-rendering strategy that warps the input image to a virtual view and then warps it back to the original view, finally resulting in an image containing holes. A refinement network is trained to refine the final image to match the original image.…”

“…We first use the learned semantic field to render multi-view semantic masks which are then translated into images using SPADE. The perceptual loss [30] Methods FID↓ KID↓ SPADE [52]+InfiniteNatureZero [36] 149.80 0.080 SPADE [52]+3DPhoto [63] 127.74 0.064 SPADE [52]+AdaMPI [20] 185.96 0.115 GVS * [19] 141.64 0.084 Ours 109.85 0.050…”

“…Although the existing single-view view synthesis methods [35,38,51,58,74,79] achieve impressive rendering results, they typically require training networks on posed multi-view images. Compared to urban or indoor scenes, learning to synthesize natural scenes is a challenging task as it is difficult to collect 3D data or posed videos of natural scenes for training, as demonstrated in [36], making the aforementioned methods not applicable. AdaMPI [20] designs a training strategy to learn the view synthesis network on single-view image collections.…”

InfiniCity: Infinite-Scale City Synthesis

“…The inference process can either be autoregressive [12,45] or non-autoregressive [22,35]. Another branch of work focuses on the long-range generation of novel views along with a camera trajectory [20,23]. However, the videos rendered by such an approach lack 3D consistency, since the appearances are modeled in the 2D image space without utilizing the 3D representations.…”