Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelines

Abstract

Figure 1. Top left: Given either real or synthetic input images, our automatic data curation allows us to find similar pose and expressions from an in-the-wild image dataset. This allows us to train personalized networks in a scalable way, requiring only a few hundred to a few thousand in-the-wild images collected via cellphones, webcams, or youtube videos. Top middle: The ability to inference from synthetic to real is used in our appearance capture pipeline. Top right: The ability to inference from real to synthetic is used in our motion tracking pipeline. Middle row: Our appearance capture pipeline, along with some representative results (on various races). Third row: Our motion capture pipeline, illustrating results from both inverse rendering and regression.