Introducing AmuS: The Amused Speech Database

Haddad, Kevin El; Torre, Ilaria; Gilmartin, Emer; Çakmak, Hüseyin; Dupont, Stéphane; Dutoit, Thierry; Campbell, Nick

doi:10.1007/978-3-319-68456-7_19

Cited by 22 publications

(22 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We manipulated the agent's emotional expressivity in terms of smiling: the agent was either congruently smiling in the voice and face, congruently neutral in the voice and face, or incongruently smiling only in the voice or only in the face. We focused on smiling as it is a universal [Mehu and Dunbar 2008] and multimodal [Eisenbarth and Alpers 2011] emotional expression, which is also audible from the voice alone [El Haddad et al 2017], and which has been shown to increase credibility and trust [Elkins and Derrick 2013;Reed et al 2018].…”

Section: Introductionmentioning

confidence: 99%

The Effect of Multimodal Emotional Expression and Agent Appearance on Trust in Human-Agent Interaction

Torre

Carrigan

McDonnell

et al. 2019

Motion, Interaction and Games

Self Cite

View full text Add to dashboard Cite

Emotional expressivity can boost trust in human-human and humanmachine interaction. As a multimodal phenomenon, previous research argued that a mismatch in the expressive channels provides evidence of joint audio-video emotional processing. However, while previous work studied this from the point of view of emotion recognition and processing, not much is known about what effect a multimodal agent would have on a human-agent interaction task. Also, agent appearance could influence this interaction too. Here we manipulated the agent's multimodal emotional expression ("smiling face" and "smiling voice", or both) and agent type (photorealistic or cartoon-like virtual human) and assessed people's trust toward this agent. We measured trust using a mixed-methods approach, combining behavioural data from a survival task, questionnaire ratings and qualitative comments. These methods gave different results: while people commented on the importance of emotional expressivity in the agent's voice, this factor had limited influence on trusting behaviours; while people rated the cartoon-like agent on several traits higher than the photorealistic one, the agent's style also was not the most influential feature on people's trusting behaviour. These results highlight the contribution of a mixedmethods approach in human-machine interaction, as both explicit and implicit perception and behaviour will contribute to the success of the interaction.

show abstract

Section: Introductionmentioning

confidence: 99%

The Effect of Multimodal Emotional Expression and Agent Appearance on Trust in Human-Agent Interaction

Torre

Carrigan

McDonnell

et al. 2019

Motion, Interaction and Games

Self Cite

View full text Add to dashboard Cite

show abstract

“…Accordingly, in [34], smiling was found associated with an increase of the second formant (F 2 ) for words with the round vowel /o:/, of intensity as well as F 0 . In [37], smiling was associated with an increase of F 2 , in [21], [38] with an increase of formants and F 0 . Higher F1 and F2 dispersion are also reported [39].…”

Section: The Acoustics Of Smiled Speechmentioning

confidence: 99%

Realistic Transformation of Facial and Vocal Smiles in Real-Time Audiovisual Streams

Arias

Soladié

Bouafif

et al. 2020

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Research in affective computing and cognitive science has shown the importance of emotional facial and vocal expressions during human-computer and human-human interactions. But, while models exist to control the display and interactive dynamics of emotional expressions, such as smiles, in embodied agents, these techniques can not be applied to video interactions between humans. In this work, we propose an audiovisual smile transformation algorithm able to manipulate an incoming video stream in real-time to parametrically control the amount of smile seen on the user's face and heard in their voice, while preserving other characteristics such as the user's identity or the timing and content of the interaction. The transformation is composed of separate audio and visual pipelines, both based on a warping technique informed by real-time detection of audio and visual landmarks. Taken together, these two parts constitute a unique audiovisual algorithm which, in addition to providing simultaneous real-time transformations of a real person's face and voice, allows to investigate the integration of both modalities of smiles in real-world social interactions.

show abstract

“…Other datasets come from research projects. Some are designed for speech generation purpose (voice conversion [19], speech synthesis [20]- [22]) while others are designed for a speech analysis goal (ASR, emotion in speech etc.). The last often consist of conversational setups and contain overlaps in speech as well as noise compared to datasets for speech generation.…”

Section: Datasetsmentioning

confidence: 99%

A Methodology for Controlling the Emotional Expressiveness in Synthetic Speech - a Deep Learning approach

Tits

2019

2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)

View full text Add to dashboard Cite

In this project, we aim to build a Text-to-Speech system able to produce speech with a controllable emotional expressiveness. We propose a methodology for solving this problem in three main steps. The first is the collection of emotional speech data. We discuss the various formats of existing datasets and their usability in speech generation. The second step is the development of a system to automatically annotate data with emotion/expressiveness features. We compare several techniques using transfer learning to extract such a representation through other tasks and propose a method to visualize and interpret the correlation between vocal and emotional features. The third step is the development of a deep learning-based system taking text and emotion/expressiveness as input and producing speech as output. We study the impact of fine tuning from a neutral TTS towards an emotional TTS in terms of intelligibility and perception of the emotion.

show abstract

Introducing AmuS: The Amused Speech Database

Cited by 22 publications

References 36 publications

The Effect of Multimodal Emotional Expression and Agent Appearance on Trust in Human-Agent Interaction

The Effect of Multimodal Emotional Expression and Agent Appearance on Trust in Human-Agent Interaction

Realistic Transformation of Facial and Vocal Smiles in Real-Time Audiovisual Streams

A Methodology for Controlling the Emotional Expressiveness in Synthetic Speech - a Deep Learning approach

Contact Info

Product

Resources

About