Emotional remapping of music to facial animation

Abstract

We propose a method to extract the emotional data from a piece of music and then use that data via a remapping algorithm to automatically animate an emotional 3D face sequence. The method is based on studies of the emotional aspect of music and our parametric-based behavioral head model for face animation. We address the issue of affective communication remapping in general, i.e. translation of affective content (eg. emotions, and mood) from one communication form to another. We report on the results of our MusicFace system, which use these techniques to automatically create emotional facial animations from multiinstrument polyphonic music scores in MIDI format and a remapping rule set.

Exploring the Cognitive Correlates of Artistic Practice Using a Parameterized Non-Photorealistic Toolkit

“…Our portrait painter research comes out of work we have explored with multidimensional parameter spaces for evolutionary generative systems (DiPaola & Gabora, 2009;DiPaola et al, 2013) and computer facial systems (DiPaola & Arya, 2006, 2007aDiPaola, 1991DiPaola, , 2002.…”

Face, portrait, mask – using a parameterised system to explore synthetic face space

“…For instance --add this expressive audio sequence to this cartoon face type but add more goofiness with a little heroicness to the personality as it animates through the given sequence outputting it as a 3D rendered movie; now take that same 'knowledge' sequence and try it on a realistic face with a more angry tone outputting it as image based movie. Also because the multidimensional parameters are align in a face centric way, it is also possible to affect a face from another knowledge or expression data stream, for instance one application of FaceSpace has been to remap emotional channels of music to emotional aspects of the face (DiPaola & Arya, 2004Arya, & 2006. Newer extensions to the parameter space approach include depiction controls which allow fine art painterly portrait depiction, which is aligned both with artistic painterly knowledge (the cognitive painterly process) as well as knowledge from vision and perception science (how humans see and perceive).…”

“…Music-driven Emotionally Expressive Face (MusicFace) (DiPaola & Arya, 2004Arya, & 2006 is a multimedia application based on our facial systems to demonstrate the concept of Affective Communication Remapping, i.e. transforming emotional information from one communication medium to another.…”