Real-time 3D Face-Eye Performance Capture of a Person Wearing VR Headset

Abstract

Teleconference or telepresence based on virtual reality (VR) headmount display (HMD) device is a very interesting and promising application since HMD can provide immersive feelings for users. However, in order to facilitate face-to-face communications for HMD users, real-time 3D facial performance capture of a person wearing HMD is needed, which is a very challenging task due to the large occlusion caused by HMD. The existing limited solutions are very complex either in setting or in approach as well as lacking the performance capture of 3D eye gaze movement. In this paper, we propose a convolutional neural network (CNN) based solution for real-time 3D face-eye performance capture of HMD users without complex modification to devices. To address the issue of lacking training data, we generate massive pairs of HMD face-label dataset by data synthesis as well as collecting VR-IR eye dataset from multiple subjects. Then, we train a dense-fitting network for facial region and an eye gaze network to regress 3D eye model parameters. Extensive experimental results demonstrate that our system can efficiently and effectively produce in real time a vivid personalized 3D avatar with the correct identity, pose, expression and eye motion corresponding to the HMD user.

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction

“…Zhang et al [16] propose a semantic volumetric representation and use it to detect 3D landmarks from an image. Song et al [17] propose a coupled radial basis function network (C-RBF) to recover the mapping between 2D and 3D faces. Cao et al [18] build a database containing 150 subjects each with 47 facial expressions, and propose a multi-linear fitting method to reconstruct 3D faces from 2D feature points.…”

“…However, they either do not publish implementation details or need specific equipment. Thies et al [55] and Song et al [17] propose a novel system to reconstruct 3D facial animation and 3D eye gaze jointly. These two methods need to attach the ArUco AR markers on the HMDs, and put the camera in the front of the user to track the HMDs which limits the range of the users' movement.…”

Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras

“…Instead of using a mouth facing camera attached to the HMD, Song et al [2018] presented a CNN based solution for real-time 3D face-eye performance with the camera looking at the whole face in non-head mounted configuration. A specific dataset was generated with labeled HMD face images and infrared eye images from multiple subjects.…”

Recovering facial reflectance and geometry from multi-view images

“…This paper considers the problem of recovering facial reflectance and geometry from images with emphasis on inferring specular albedo, which is important as real faces often exhibit bright highlights due to specular reflection. Accurately recovering specular components facilitates more realistic rendering of digital faces in new environments, which is very useful in many applications such as augmented and virtual reality, games, entertainment [16,23,30] and telepresence [6,12,20,33]. It also helps in the recovery of highfidelity 3D geometry for faces, which is a fundamental problem in computer vision and graphics.…”