Research in affective computing and cognitive science has shown the importance of emotional facial and vocal expressions during human-computer and human-human interactions. But, while models exist to control the display and interactive dynamics of emotional expressions, such as smiles, in embodied agents, these techniques can not be applied to video interactions between humans. In this work, we propose an audiovisual smile transformation algorithm able to manipulate an incoming video stream in real-time to parametrically control the amount of smile seen on the user's face and heard in their voice, while preserving other characteristics such as the user's identity or the timing and content of the interaction. The transformation is composed of separate audio and visual pipelines, both based on a warping technique informed by real-time detection of audio and visual landmarks. Taken together, these two parts constitute a unique audiovisual algorithm which, in addition to providing simultaneous real-time transformations of a real person's face and voice, allows to investigate the integration of both modalities of smiles in real-world social interactions.

Reconstructing the geometric structure of a face from a single input image is a challenging active research area in computer vision. In this paper, we present a novel method for reconstructing 3D heads from an input image using a hybrid approach based on learning and geometric techniques. We introduce a deep neural network trained on synthetic data only, which predicts the map of normal vectors of the face surface from a single photo. Afterward, using the network output we recover the 3D facial geometry by means of weighted least squares. Through qualitative and quantitative evaluation tests, we show the accuracy and robustness of our proposed method. Our method does not require accurate alignment due to the image-to-image translation network and also successfully recovers 3D geometry for real images, despite the fact that the model was trained only on synthetic data.

No abstract