Automatic Recognition of Affective Laughter in Spontaneous Dyadic Interactions from Audiovisual Signals

Kantharaju, Reshmashree B.; Ringeval, Fabien; Besacier, Laurent

doi:10.1145/3242969.3243012

Cited by 12 publications

(4 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interesting research about the role of laughter in conversational dynamics (e.g. turn-taking, change of topic, end of conversation) and its sequential patterns of occurrence and acoustic characteristics is available and proliferating (among others [12], [13], [14], [15]). Nevertheless, little attention has been devoted to identifying what laughter is about, and most researchers takes as given the assumption that laughter is adjacent to what it is related to.…”

Section: Introductionmentioning

confidence: 99%

What’s Your Laughter Doing There? A Taxonomy of the Pragmatic Functions of Laughter

Mazzocconi

Tian²,

Ginzburg

2022

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

L AUGHTER is a crucial signal for communication and managing interactions. Until now no consensual approach has emerged for classifying laughter. We propose a new framework for laughter analysis and classification, based on the pivotal assumption that laughter has propositional content. We propose an annotation scheme to classify the pragmatic functions of laughter taking into account the form, the laughable, the social, situational, and linguistic context. We apply the framework and taxonomy proposed in a multilingual corpus study (French, Mandarin Chinese and English), involving a variety of situational contexts. Our results give rise to novel generalizations about the range of meanings laughter exhibits, the placement of the laughable, and how placement and arousal relate to the functions of laughter. We have tested and refuted the validity of the commonly accepted assumption that laughter directly follows its laughable. In the concluding section, we discuss the implications our work has for spoken dialogue systems. We stress that laughter integration in spoken dialogue systems is not only crucial for emotional and affective computing aspects, but also for aspects related to natural language understanding and pragmatic reasoning. We formulate the emergent computational challenges for incorporating laughter in spoken dialogue systems.

show abstract

Section: Introductionmentioning

confidence: 99%

What’s Your Laughter Doing There? A Taxonomy of the Pragmatic Functions of Laughter

Mazzocconi

Tian²,

Ginzburg

2022

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

show abstract

“…The authors' best model achieved UAR (Arousal)=60.77% and UAR (Valence)=52.3% using RECOLA's samples. Kantharaju et al [50] used facial action units (FAUs) and audiovisual signals to classify negative and positive emotions. Their emotion detection samples were filtered based on laughter episodes.…”

Section: B Results 1) Classifier Performancementioning

confidence: 99%

“…Corresponding to the time of the video clip, each physiological signal window size of RECOLA is 160. The DBN is constructed with hidden layers sized [160, 1000, 500,110] and fully connected layers sized [110, 50,45]. The training process contains pretraining and fine-tuning.…”

Section: ) Physiological Featurementioning

confidence: 99%

Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

Xie

Lv³

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The Internet of Things (IoT) technologies such as interconnection and edge computing help emotion recognition to be applied in healthcare, smart education, etc. However, the acquisition and transmission processes may have some situations, such as lost signals and serious interference noise caused by motion, which affect the quality of the received data and limit the performance of IoT emotion detection. We collectively refer to these as invalid data. A multi-step deep (MSD) system is proposed to reliably detect multimodal emotion by the collected records containing invalid data. Semantic compatibility and continuity are utilized to filter out the invalid data. The feature from invalid modal data is replaced through the imputation method to compensate for the impact of invalid data on emotion detection. In this way, the proposed system can automatically process invalid data and improve the recognition performance. Furthermore, considering the spatiotemporal information, the features of video and physiological signals are extracted by specific deep neural networks in the MSD system. The simulation experiments are conducted on a public multimodal database, and the performance of the MSD system measured by the unweighted average recall is better than that of the traditional system. The promising results observed in the experiments verify the potential influence of the proposed system in practical IoT applications. INDEX TERMS Internet of Things, multimodal emotion detection, invalid data, multi-step deep (MSD) system, deep neural networks

show abstract

“…From an intelligent systems perspective, creating models that do automatic laughter detection is arguably the most common task using both audio and visual features for training ( Truong and Van Leeuwen, 2007 ; Cosentino et al, 2016 ; Turker et al, 2017 ; Akhtar et al, 2018 ; Kantharaju et al, 2018 ; Ataollahi and Suarez, 2019 ; Gosztolya and Tóth, 2019 ), and ubiquitous devices such as computer microphones and web cameras can provide reasonably accurate detection. These studies primarily distinguish between speech and laughter for an inter-pausal unit (IPU) or perform detection using a continuous model.…”

Section: Related Workmentioning

confidence: 99%

Can a robot laugh with you?: Shared laughter generation for empathetic spoken dialogue

2022

View full text Add to dashboard Cite

Spoken dialogue systems must be able to express empathy to achieve natural interaction with human users. However, laughter generation requires a high level of dialogue understanding. Thus, implementing laughter in existing systems, such as in conversational robots, has been challenging. As a first step toward solving this problem, rather than generating laughter from user dialogue, we focus on “shared laughter,” where a user laughs using either solo or speech laughs (initial laugh), and the system laughs in turn (response laugh). The proposed system consists of three models: 1) initial laugh detection, 2) shared laughter prediction, and 3) laugh type selection. We trained each model using a human-robot speed dating dialogue corpus. For the first model, a recurrent neural network was applied, and the detection performance achieved an F1 score of 82.6%. The second model used the acoustic and prosodic features of the initial laugh and achieved a prediction accuracy above that of the random prediction. The third model selects the type of system’s response laugh as social or mirthful laugh based on the same features of the initial laugh. We then implemented the full shared laughter generation system in an attentive listening dialogue system and conducted a dialogue listening experiment. The proposed system improved the impression of the dialogue system such as empathy perception compared to a naive baseline without laughter and a reactive system that always responded with only social laughs. We propose that our system can be used for situated robot interaction and also emphasize the need for integrating proper empathetic laughs into conversational robots and agents.

show abstract

Automatic Recognition of Affective Laughter in Spontaneous Dyadic Interactions from Audiovisual Signals

Cited by 12 publications

References 44 publications

What’s Your Laughter Doing There? A Taxonomy of the Pragmatic Functions of Laughter

What’s Your Laughter Doing There? A Taxonomy of the Pragmatic Functions of Laughter

Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

Can a robot laugh with you?: Shared laughter generation for empathetic spoken dialogue

Contact Info

Product

Resources

About