StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

A Survey on 3D-aware Image Synthesis

“…Shape Quality is evaluated mostly by calculating differences between the rendered depth map and the pseudoground-truth depth, e.g. using MSE [48] or a modified Chamfer distance [50]. For example, given two generated images from two sampled angles of the same scene, Shi et al [52] uses rotation precision and rotation consistency to evaluate the quality of the depth maps (point cloud).…”

“…Due to the prevalence of 2D generative models, there have been studies aiming to make these pretrained models, especially GANs, 3D aware. These works, mostly built on the [48], [50], [51] single simple-shape AFHQ [82] CVPR 2020 Cat, Dog, and Wildlife 15k 512 × 512 [1], [48], [50], [51] single simple-shape CompCars [72] CVPR 2015…”

“…HoloGAN [16] ICCV 2019 face, cat, car, LSUN camera pose unsupervised deep voxel Liao et al [42] CVPR 2020 multiple object data unconditional unsupervised 3D primitives BlockGAN [20] NeurIPS 2020 multiple object data camera pose unsupervised deep voxel GRAF [28] NeurIPS 2020 rendered chair, face, cat, bird camera matrix, pose unsupervised NeRF pi-GAN [2] CVPR 2021 face, car, CARLA camera position unsupervised NeRF GIRAFFE [32] CVPR 2021 chair, cat, face, car, church camera pose unsupervised compositional NeRF GOF [45] NeurIPS 2021 cat, car, face 3D location, camera pose unsupervised generative occupancy fields ShadeGAN [43] NeurIPS 2021 cat, face 3D location, camera pose unsupervised light field CAMPARI [39] 3DV 2021 cat, car, face, chair camera pose unsupervised decomposed NeRF StyleNeRF [1] ICLR 2022 face, cat, car camera pose unsupervised NeRF GRAM [49] CVPR 2022 cat, face, CARLA camera pose unsupervised EG3D [48] CVPR 2022 cat, face camera parameters unsupervised tri-plane 3D representation VolumeGAN [51] CVPR 2022 cat, car, face, bedroom, CARLA camera pose unsupervised StyleSDF [50] CVPR 2022 cat, face camera pose unsupervised SDF Pix2NeRF [71] CVPR 2022 face, CARLA, rendered image image unsupervised Sem2NeRF [57] ECCV 2022 cat, face semantic mask, camera pose semantic mask and image SURF-GAN [63] ECCV 2022 face camera pose unsupervised EpiGRAF [60] NeurIPS 2022 cat, face, variable-shape camera pose unsupervised tri-plane 3D representation IDE-3D [61] TOG 2022 face camera pose semantic mask and image tri-plane 3D representation control over the 3D parameters θ gives the edited result x . Here, θ could be human-interpretable attribute descriptions or a set of parameters from 3D models.…”

“…A neural renderer is finally used for appearance synthesis. StyleSDF [50] (Dec 2021) merges a SDF-based 3D representation into the 2D StyleGAN generator. This framework consists of two main components: a backbone conditional SDF volume renderer and a StyleGAN generator.…”

“…Due to the high memory and computation cost of volume rendering, direct rendering at target resolution is not efficient and struggles to generate images at high-resolution (512×512 and beyond). Some recently-developed methods make use of a two-stage rendering process [1], [32], [48], [50], [51] or develop efficient volume rendering strategy [43], [44], [49] for high-resolution image generation. Meanwhile, aimed to reduce view-inconsistent artifacts brought by the 2D renderers, they adopt different strategies such as NeRF path regularization [1] and dual discriminators [48].…”