Smart sensor integration: A framework for multimodal emotion recognition in real-time

Wagner, Johannes; André, Elisabeth; Jung, Frank

doi:10.1109/acii.2009.5349571

Cited by 38 publications

(23 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Section 2 is dedicated to the software used during this project: Smart Sensor Integration (SSI) [39] for recording, annotating and analyzing laughs; MediaCycle [27] to compute similarities between laughs; Greta [16] for playing the output laughter. Section 3 presents the audiovisual laughter database, that contains laughs used to animate Greta.…”

Section: Motivation and Related Workmentioning

confidence: 99%

AVLaughterCycle

Urbain

Niewiadomski

Bevacqua

et al. 2010

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

The AVLaughterCycle project aims at developing an audiovisual laughing machine, able to detect and respond to user's laughs. Laughter is an important cue to reinforce the engagement in human-computer interactions. As a first step toward this goal, we have implemented a system capable of recording the laugh of a user and responding to it with a similar laugh. The output laugh is automatically selected from an audiovisual laughter database by analyzing acoustic similarities with the input laugh. It is displayed by an Embodied Conversational Agent, animated using the audio-synchronized facial movements of the subject who originally uttered the laugh. The application is fully implemented, works in real time and a large audiovisual laughter database has been recorded as part of the project. This paper presents AVLaughterCycle, its underlying components, the freely available laughter database and the application architecture. The paper also includes evaluations of several core components of the application. Objective tests show that the similarity search engine, though simple, significantly outperforms chance for grouping laughs Portions of this work have been presented in "Proceedings of eNTERFACE'09" [36]. by speaker or type. This result can be considered as a first measurement for computing acoustic similarities between laughs. A subjective evaluation has also been conducted to measure the influence of the visual cues on the users' evaluation of similarity between laughs.

show abstract

Section: Motivation and Related Workmentioning

confidence: 99%

AVLaughterCycle

Urbain

Niewiadomski

Bevacqua

et al. 2010

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…This design preference was motivated by a wish to ensure minimal latency between a change in the user's environment and the time at which the agent is aware of the change. However, there have been recent advances in realtime processing of environmental data as a means of determining affective state, such as Wagner's smart sensor integration (SSI) framework [40]. The SSI framework supports realtime affect recognition from sensors and can output the resulting affective state in a variety of formats including a format similar to our emotion ontology.…”

Section: Sensorsmentioning

confidence: 99%

Koko: an architecture for affect-aware games

Sollenberger

Singh

2010

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

The importance of affect in delivering engaging experiences in entertainment and educational games is well recognized. Yet, current techniques for building affect-aware games are limited, with the maintenance and use of affect in essence being handcrafted for each game. The Koko architecture describes a service-oriented middleware that reduces the burden of incorporating affect recognition into games, thereby enabling developers to concentrate on the functional and creative aspects of their applications. The Koko architecture makes three key contributions: (1) improving developer productivity by creating a reusable and extensible environment; (2) yielding an enhanced user experience by enabling independently developed games and other applications to collaborate and provide a more coherent user experience than currently possible; (3) enabling affective communication in multiplayer and social games. Further, Koko is intended to be used as an extension of existing game architectures. We recognize that complex games require additional third party libraries, such as game engines. To enable the required flexibility we define the interfaces of the Koko architecture in a formal manner, thereby enabling the implementation of those interfaces to readily adapt to the unique requirements of game's other architectural components and requirements.

show abstract

“…Note that described methods can be plugged to a recognition pipeline that processes live input from a single or multiple sensor devices. SSI automatically handles synchronization between the components of a pipeline and allows several pipelines to run in parallel and share data streams [28].…”

Section: Affective Recognitionmentioning

confidence: 99%

“…In the following, we present our Social Signal Interpretation (SSI) tool, a framework dedicated to support the development of online recognition systems [28]. SSI covers the tasks necessary to assemble a complete machine learning pipeline, ranging from live sensor input and real-time signal processing, to model training and online classification.…”

mentioning

confidence: 99%

Social Signal Interpretation (SSI)

et al. 2011

Self Cite

View full text Add to dashboard Cite

The development of anticipatory user interfaces is a key issue in human-centred computing. Building systems that allow humans to communicate with a machine in the same natural and intuitive way as they would with each other requires detection and interpretation of the user's affective and social signals. These are expressed in various and often complementary ways, including gestures, speech, mimics etc. Implementing fast and robust recognition engines is not only a necessary, but also challenging task. In this article, we introduce our Social Signal Interpretation (SSI) tool, a framework dedicated to support the development of such online recognition systems. The paper at hand discusses the processing of four modalities, namely audio, video, gesture and biosignals, with focus on affect recognition, and explains various approaches to fuse the extracted information to a final decision.

show abstract

Smart sensor integration: A framework for multimodal emotion recognition in real-time

Cited by 38 publications

References 13 publications

AVLaughterCycle

AVLaughterCycle

Koko: an architecture for affect-aware games

Social Signal Interpretation (SSI)

Contact Info

Product

Resources

About