PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

Abstract

We present PVSeRF, a learning framework that reconstructs neural radiance fields from single-view RGB images, for novel view synthesis. Previous solutions, such as pix-elNeRF [66], rely only on pixel-aligned features and suffer from feature ambiguity issues. As a result, they struggle with the disentanglement of geometry and appearance, leading to implausible geometries and blurry results. To address this challenge, we propose to incorporate explicit geometry reasoning and combine it with pixel-aligned features for radiance field prediction. Specifically, in addition to pixel-aligned features, we further constrain the radiance field learning to be conditioned on i) voxel-aligned features learned from a coarse volumetric grid and ii) fine surfacealigned features extracted from a regressed point cloud. We show that the introduction of such geometry-aware features helps to achieve a better disentanglement between appearance and geometry, i.e. recovering more accurate geometries and synthesizing higher quality images of novel views. Extensive experiments against state-of-the-art methods on ShapeNet benchmarks demonstrate the superiority of our approach for single-image novel view synthesis.

MMNeRF: Multi-Modal and Multi-View Optimized Cross-Scene Neural Radiance Fields

“…As seen in Table 2, the results of our experiments are satisfactory. We achieve significant performance improvements on the strong baseline SRN [23], DVR [25], pixelNeRF [7] and PVSeRF [13], which further confirmed our model generalizes well to novel scenes. The last column of Table 2 lists the mean performance of our methods on 13 categories.…”

“…To compensate for the lack of spatial information, PVSeRF [13] proposes to incorporate explicit geometry reasoning, i.e., coarse voxel-aligned and fine surface-aligned features, and combine them with pixel-aligned features for radiance field prediction. Introducing such geometry-aware features helps to achieve a better disentanglement between appearance and geometry.…”