An extensible framework for interactive facial animation with facial expressions, lip synchronization and eye behavior

Abstract

In this article we describe our approach to generating convincing and empathetic facial animation. Our goal is to develop a robust facial animation platform that is usable and can be easily extended. We also want to facilitate the integration of research in the area and to directly incorporate the characters in interactive applications such as embodied conversational agents and games. We have developed a framework capable of easily animating MPEG-4 parameterized faces through highlevel description of facial actions and behaviors. The animations can be generated in real time for interactive applications. We present some case studies that integrate computer vision techniques in order to provide interaction between the user and a character that interacts with different facial actions according to events in the application.

VhCVE: A Collaborative Virtual Environment Including Facial Animation and Computer Vision

“…It explores the functionalities of the VHFace and Webcam Manager modules, using facial feature detector algorithms (for face, eye and mouth regions) from OpenCV [Viola and Jones 2001;Castrillón et al 2007]. The mapping of the detected features to simple events (opened mouth, closed mouth, smile and direction of the horizontal gaze) and then to FAP animations is implemented according to the methodology of Queiroz et al [Queiroz et al 2009]. Once information from face components is acquired, it is used to animate the avatar's face, as shown in Figures 1 and 11.…”

“…The facial animation module, called VHFace Manager, integrates the framework proposed by Queiroz et al [Queiroz et al 2009], based on XFace core libraries [Balci 2004]. The framework follows the MPEG-4 Facial Animation (FA) standard [Pandzic and Forchheimer 2003] for parameterization and animation of faces.…”

“…• The CV2FAP/FDL2FAP module, which receives as input Computer Vision data or high-level face actions and maps it to FAPs, according to Queiroz et al [Queiroz et al 2009] methodology. The high-level face actions include facial expressions, lip synchronization and eye behaviors, using the FDL script language (Facial Description Language), also described in [Queiroz et al 2009].…”

“…The high-level face actions include facial expressions, lip synchronization and eye behaviors, using the FDL script language (Facial Description Language), also described in [Queiroz et al 2009]. …”

Lip syncing method for realistic expressive 3D face model

“…Expressive visual speech being the emotional manifestation of a personality requires efficient and pragmatic communication. Advanced research on facial animation mainly focussed to understanding clearly the animated avatar that explores the expressiveness, communication and interactivity aspects of ECA development [31]. Human face being the mirror of internal sensation is undoubtedly the most significant art object and pivotal part that plays a meaningful role in this interaction process.…”