LOLNeRF: Learn from One Look

Rebain, Daniel; Matthews, Mark W.; Yi, Kwang Moo; Lagun, Dmitry; Tagliasacchi, Andrea

doi:10.1109/cvpr52688.2022.00161

Cited by 74 publications

(20 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An example of synthetic data rendering process is to center a scan (e.g., objects from ScanNet [66], ShapeNet [67], or DeepVoxels [14]) at the origin, scale it to lie within the unit cube, and render images at sampled viewpoints. The training set can be obtained by 2019 2020 2021 2022 2023 S 2 -GAN [12] PrGAN [13] DeepVoxel [14] VON [15] HoloGAN [16] SRN [17] GANSteerability [18] RGBD-GAN [19] DVR [7] BlockGAN [20] GANLatentDiscovery [21] NeRF [5] StyleRig [22] CONFIG [23] GANSpace [24]) InterFaceGAN [25]) NGP [26] SeFa [27] GRAF [28] NeRF-W [29] PIE [30] NeRF++ [31] GIRAFFE [32] pi-GAN [2] PixelNeRF [33] GAN-Control [34] NeRF- [35] KiloNeRF [36] Mip-NeRF [37] FastNeRF [38] CAMPARI [39] BARF [40] VariTex [41] Liao et al [42] ShadeGAN [43] CIPS-3D [44] StyleNeRF [1] GOF [45] LOLNeRF [46] URF [47] EG3D [48] GRAM [49] StyleSDF ...…”

Section: Multiple-view Image Collectionsmentioning

confidence: 99%

“…Human Face 30k 1024 × 1024 [46] single, simple-shape MetFaces [83] NeurIPS 2020 Art Face 1336 1024 × 1024 [1] single, simple-shape M-Plants [60] NeurIPS 2022 Variable-Shape 141,824 256 × 256 [60] single, variable-shape M-Food [60] NeurIPS 2022 Variable-Shape 25,472 256 × 256 [60] single, variable-shape top of a pretrained StyleGAN, can be further categorized into three groups based on how the 3D control capability is introduced: 1) discovering 3D control latent directions, 2) adopting explicit control over the 3D parameters, and 3) introducing 3D-aware components into 2D GANs. Fig.…”

Section: D Control Of 2d Generative Modelsmentioning

confidence: 99%

“…The majority of methods in this part belong to the category of GANs. Some use generative latent optimization [46], [138] instead of adversarial training [139]. Other kinds of generative models, especially diffusion models [140], [141], which have proven extremely effective in generating high-quality images in recent years, have not yet been widely applied to 3D-ware image synthesis.…”

Section: D-aware Generative Modelsmentioning

confidence: 99%

“…A follow-up work pi-GAN [2] (Dec 2020) differs from GRAF in three ways on network architecture and training strategy: 1) pi-GAN uses SIREN [143] as the choice of scene representation rather than a positionally-encoded ReLU MLP [5]; 2) pi-GAN leverages a StyleGAN-inspired mapping network to condition layers in the SIREN on a single input noise code through feature-wise linear modulation (FiLM) instead of conditioning on two additional shape/appearance codes; 3) pi-GAN follows ProgressiveGAN [77] where discriminator grows progressively rather than a patch-based discriminator. Built similarly to pi-GAN, LOLNeRF [46] (Nov 2021), which is capable of single-shot view synthesis of human faces, uses generative latent optimization [138] instead of adversarial training [139]. GIRAFFE [32] improves the BlockGAN [20] framework by replacing the voxel-based representation and 3D-to-2D projection with a NeRF-based compositional 3D scene representation and a neural rendering pipeline.…”

Section: Efficient and Expressive Representationsmentioning

confidence: 99%

See 3 more Smart Citations

A Survey on 3D-aware Image Synthesis

Xia¹,

Xue²

2022

Preprint

View full text Add to dashboard Cite

Recent years have seen remarkable progress in deep learning powered visual content creation. This includes 3D-aware generative image synthesis, which produces high-fidelity images in a 3D-consistent manner while simultaneously capturing compact surfaces of objects from pure image collections without the need for any 3D supervision, thus bridging the gap between 2D imagery and 3D reality. The 3D-aware generative models have shown that the introduction of 3D information can lead to more controllable image generation. The task of 3D-aware image synthesis has taken the field of computer vision by storm, with hundreds of papers accepted to top-tier journals and conferences in recent year (mainly the past two years), but there lacks a comprehensive survey of this remarkable and swift progress. Our survey aims to introduce new researchers to this topic, provide a useful reference for related works, and stimulate future research directions through our discussion section. Apart from the presented papers, we aim to constantly update the latest relevant papers along with corresponding implementations at https://weihaox.github.io/awesome-3D-aware-synthesis.

show abstract

Section: Multiple-view Image Collectionsmentioning

confidence: 99%

Section: D Control Of 2d Generative Modelsmentioning

confidence: 99%

Section: D-aware Generative Modelsmentioning

confidence: 99%

Section: Efficient and Expressive Representationsmentioning

confidence: 99%

See 2 more Smart Citations

A Survey on 3D-aware Image Synthesis

Xia¹,

Xue²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…StyleNeRF and CIPS-3D [16,72] combine a shallow NeRF network to provide low-resolution radiance fields and a 2D rendering network to produce high-resolution images with fine details. LolNeRF [39] learns 3D objects by optimizing foreground and background NeRFs together with a learnable per-image table of latent codes. Zhao et al [71] develop a generative multi-plane image (GMPI) representation to ensure view-consistency.…”

Section: Related Workmentioning

confidence: 99%

Controllable Radiance Fields for Dynamic Face Synthesis

Zhuang¹,

Ma²,

Koyejo³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

KeypointNeRF: Generalizing Image-Based Volumetric Avatars Using Relative Spatial Encoding of Keypoints

Mihajlovic

Bansal²,

Zollhöfer³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Image-based volumetric humans using pixel-aligned features promise generalization to unseen poses and identities. Prior work leverages global spatial encodings and multi-view geometric consistency to reduce spatial ambiguity. However, global encodings often suffer from overfitting to the distribution of the training data, and it is difficult to learn multi-view consistent reconstruction from sparse views. In this work, we investigate common issues with existing spatial encodings and propose a simple yet highly effective approach to modeling high-fidelity volumetric humans from sparse views. One of the key ideas is to encode relative spatial 3D information via sparse 3D keypoints. This approach is robust to the sparsity of viewpoints and cross-dataset domain gap. Our approach outperforms state-of-the-art methods for head reconstruction. On human body reconstruction for unseen subjects, we also achieve performance comparable to prior work that uses a parametric human body model and temporal feature aggregation. Our experiments show that a majority of errors in prior work stem from an inappropriate choice of spatial encoding and thus we suggest a new direction for high-fidelity image-based human modeling.

show abstract

LOLNeRF: Learn from One Look

Cited by 74 publications

References 46 publications

A Survey on 3D-aware Image Synthesis

A Survey on 3D-aware Image Synthesis

Controllable Radiance Fields for Dynamic Face Synthesis

KeypointNeRF: Generalizing Image-Based Volumetric Avatars Using Relative Spatial Encoding of Keypoints

Contact Info

Product

Resources

About