Generating Facial Expressions for Speech

Abstract

This paper reports results from a program that produces high quality animation of facial expressions and head movements as automatically as possible in conjunction with meaning-based speech synthesis, including spoken intonation. The goal of the research is as much to test and define our theories of the formal semantics for such gestures, as to produce convincing animation. Towards this end we have produced a high level programming language for 3D animation of facial expressions. We have been concerned primarily with expressions conveying information correlated with the intonation of the voice: this includes the differences of timing, pitch, and emphasis that are related to such semantic distinctions of discourse as "focus", "topic" and "comment", "theme" and "rheme", or "given" and "new" information. We are also interested in the relation of affect or emotion to facial expression. Until now, systems have not embodied such rule-governed translation from spoken utterance meaning to facial expressions. Our system embodies rules that describe and coordinate these relations: intonation/information, intonation/affect and facial expressions/affect. A meaning representation includes discourse information: what is contrastive/background information in the given context, and what is the "topic" or "theme" of the discourse. The system maps the meaning representation into how accents and their placement are chosen, how they are conveyed over facial expression and how speech and facial expressions are coordinated. This determines a sequence of functional groups: lip shapes, conversational signals, punctuators, regulators or manipulators. Our algorithms then impose synchrony, create coarticulation effects, and determine affectual signals, eye and head movements. The lowest level representation is the Facial Action Coding System (FACS), which makes the generation system portable to other facial models. Disciplines Computer Sciences | Engineering | Graphics and Human Computer InterfacesThis journal article is available at ScholarlyCommons: http://repository.upenn.edu/hms/192 Generating Facial Expressions for Speech 1 AbstractThis paper reports results from a program that produces high quality animation of facial expressions and head movements as automatically as possible in conjunction with meaning-based speech synthesis, including spoken intonation. The goal of the research is as much to test and define our theories of the formal semantics for such gestures, as to produce convincing animation. Towards this end we have produced a high level programming language for 3D animation of facial expressions. We have been concerned primarily with expressions conveying information correlated with the intonation of the voice: this includes the differences of timing, pitch, and emphasis that are related to such semantic distinctions of discourse as "focus", "topic" and "comment", "theme" and "rheme", or "given" and "new" information. We are also interested in the relation of affect or emotion to facial expression. Until now, syste...

The Architectural Role of Emotion in Cognitive Systems

“…Classic work on virtual humans in the computer graphics community focuses on perception and action in 3D worlds (Badler, Phillips, & Webber, 1993;Thalmann, 1993), but largely ignores dialogue and emotions. Several systems have carefully modeled the interplay between speech and nonverbal behavior in face-to-face dialogue (Cassell, Bickmore, Campbell, Vilhjálmsson, & Yan, 2000;Pelachaud, Badler, & Steedman, 1996) but these virtual humans do not include emotions and can not participate in physical tasks in 3D worlds. Some work has begun to explore the integration of conversational capabilities with emotions (Lester, Towns, Callaway, Voerman, & FitzGerald, 2000;Marsella, Johnson, & LaBore, 2000;Poggi & Pelachaud, 2000), but still does not address physical tasks in 3D worlds.…”

Speech-driven facial animation with realistic dynamics

“…In textdriven facial animation, the process generally involves determining a mapping from text (orthographic or phonetic) onto visemes by means of vector quantization [10], [11] or a rulebased system [12], [13]. Facial animation driven by speech can be approached in a similar fashion by deriving the phoneme sequence directly from the speech signal, as is done in speech recognition [14]- [16].…”

Synthesizing cooperative conversation

“…Each is represented by two parameters: its time of occurrence and its type. Our algorithm [53] embodies rules to automatically generate facial expressions, following the principle of synchrony. The program scans the input utterances and computes the different facial expressions corresponding to these functional groups.…”

“…The program scans the input utterances and computes the different facial expressions corresponding to these functional groups. The computation of the lip shape is done in three passes and incorporates coarticulation effects [53]. Phonemes are characterized by their degree of deformability.…”