3D GAN Inversion with Pose Optimization

Ko, Jae-Hoon; Cho, Kyusun; Choi, Daewon; Ryoo, Kwangrok; Kim, Seungryong

doi:10.1109/wacv56688.2023.00298

Cited by 42 publications

(12 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, the ICP algorithm in our 3D alignment occasionally yields incorrect 3D alignment. These problems can be mitigated by the new advances [52,87] in 3D-aware GAN inversion and 3D alignment. See Appendix H for more discussion.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

College students’ perceptions of AI-based writing learning tools: With a focus on Google Translate, Naver Papago, and Grammarly

Kim¹,

Han²

2021

meeso

View full text Add to dashboard Cite

Image blending aims to combine multiple images seamlessly. It remains challenging for existing 2D-based methods, especially when input images are misaligned due to differences in 3D camera poses and object shapes. To tackle these issues, we propose a 3D-aware blending method using generative Neural Radiance Fields (NeRF), including two key components: 3D-aware alignment and 3D-aware blending. For 3D-aware alignment, we first estimate the camera pose of the reference image with respect to generative NeRFs and then perform 3D local alignment for each part. To further leverage 3D information of the generative NeRF, we propose 3D-aware blending that directly blends images on the NeRF's latent representation space, rather than raw pixel space. Collectively, our method outperforms existing 2D baselines, as validated by extensive quantitative and qualitative evaluations with FFHQ and AFHQ-Cat.

show abstract

Section: Discussion and Limitationsmentioning

confidence: 99%

“…Figure 17 shows another failure case of 3D-aware alignment in the Iterative Closest Point (ICP) algorithm [4,19]. ICP is an approach to aligning the two point clouds: P of inversion [52] and point-set registration techniques [87] can boost our blending performance in the future.…”

Section: H Failure Casesmentioning

confidence: 99%

College students’ perceptions of AI-based writing learning tools: With a focus on Google Translate, Naver Papago, and Grammarly

Kim¹,

Han²

2021

meeso

View full text Add to dashboard Cite

show abstract

“…3D GAN inversion approaches [49,54,70,79,89] follow the concept of 2D GAN inversion but (jointly) optimize the face latent code and part of the camera parameters to achieve 3D consistent reconstruction and manipulation. Some recent methods [54] utilize the pre-trained EG3D [19] or retrain the model [70] to manipulate a single portrait image.…”

Section: Gan Inversion Gan Inversion Maps a Real Image Backmentioning

confidence: 99%

“…Some recent methods [54] utilize the pre-trained EG3D [19] or retrain the model [70] to manipulate a single portrait image. Among these methods, [70] and [79] first estimate all camera parameters (intrinsic and extrinsic) from 3DMM and then keep them fixed; [49] jointly optimizes the face latent code and rotation of the camera, assuming known camera intrinsics and camera-to-face distance. In contrast, we develop an optimization schedule to estimate the face latent code and camera parameters jointly.…”

Section: Gan Inversion Gan Inversion Maps a Real Image Backmentioning

confidence: 99%

“…Existing 3D GAN inversions [49,70] avoid this problem since they focus on face images captured at far camera-tosubject distances, where the weak perspective model can be used to approximate the imaging process, and the depth of the face can be considered negligible. The shared face shape among different combinations of focal length and camera-to-subject distance mitigates the ill-posedness of the problem, allowing their use of large focal lengths with roughly estimated camera-to-subject distances.…”

Section: Perspective-aware 3d Gan Inversionmentioning

confidence: 99%

See 1 more Smart Citation