Performance-Driven Facial Animation: Basic Research on Human Judgments of Emotional State in Facial Avatars

Abstract

Virtual reality is rapidly evolving into a pragmatically usable technology for mental health (MH) applications. As the underlying enabling technologies continue to evolve and allow us to design more useful and usable structural virtual environments (VEs), the next important challenge will involve populating these environments with virtual representations of humans (avatars). This will be vital to create mental health VEs that leverage the use of avatars for applications that require human-human interaction and communication. As Alessi et al. 1 pointed out at the 8th Annual Medicine Meets Virtual Reality Conference (MMVR8), virtual humans have mainly appeared in MH applications to "serve the role of props, rather than humans." More believable avatars inhabiting VEs would open up possibilities for MH applications that address social interaction, communication, instruction, assessment, and rehabilitation issues. They could also serve to enhance realism that might in turn promote the experience of presence in VR. Additionally, it will soon be possible to use computer-generated avatars that serve to provide believable dynamic facial and bodily representations of individuals communicating from a distance in real time. This could support the delivery, in shared virtual environments, of more natural human interaction styles, similar to what is used in real life between people. These techniques could enhance communication and interaction by leveraging our natural sensing and perceiving capabilities and offer the potential to model human-computer-human interaction after human-human interaction. To enhance the authenticity of virtual human representations, advances in the rendering of facial and gestural behaviors that support implicit communication will be needed. In this regard, the current paper presents data from a study that compared human raters' judgments of emotional expression between actual video clips of facial expressions and identical expressions rendered on a three-dimensional avatar using a performance-driven facial animation (PDFA) system developed at the University of Southern California Integrated Media Systems Center. PDFA offers a means for creating high-fidelity visual representations of human faces and bodies. This effort explores the feasibility of sensing and reproducing a range of facial expressions with a PDFA system. In order to test concordance of human ratings of emotional expression between video and avatar facial delivery, we first had facial model subjects observe stimuli that were designed to elicit naturalistic facial expressions. The emotional stimulus induction involved presenting text-based, still image, and video clips to subjects that were previously rated to induce facial expressions for the six universals 2 of facial expression (happy, sad, fear, anger,

Analysis of assets for virtual reality applications in neuropsychology

“…This has been mainly due to challenges for both the creation of avatars that can dynamically communicate non-verbal implicit signals via facial and body gestures and in the capacity to drive such avatar expression/interaction with some form of artificial intelligence. Research on these issues is actually quite active from a basic science perspective (Rickel, Marsala, Gratch, Hill, Traum, & Swartout, 2002;Rizzo et al, 2001a), but high development costs and technical challenges have thus far limited progress for all but the most basic direct clinical applications.…”

Evaluating the perceptual realism of animated facial expressions

“…Their evaluation thus relied on an indirect, perceptual measure rather than on an explicit judgment of realism. Another example of a perceptual investigation of avatars is the work by Rizzo et al [2001], in which recognition of 3D animated facial expressions was compared with recognition of video sequences. On average, their avatar performed consistently worse than the corresponding video sequences; unfortunately, they used different actors for each facial expression, which makes generalizations from the experimental results difficult.…”

Psychophysical evaluation of animated facial expressions

“…These measures allow us to characterize the perception of facial expressions at different levels: At the most basic level, expressions of computer animated faces have to be recognizable (see Cunningham et al 2004;Rizzo et al 2001]). We have therefore chosen recognition as one of the central perceptual tasks in the following experiments.…”