Audiovisual Detection of Laughter in Human-Machine Interaction

Abstract

Abstract-Laughter is clearly an audiovisual event, consisting of the laughter vocalization and of facial activity, mainly around the mouth and sometimes in the upper face. However, past research on laughter recognition has mainly focused on the information available in the audio channel only, mainly due to the lack of suitable audiovisual data. Only recently few works have been published which combine audio and visual information and most of them deal with the problem of discriminating laughter from speech or other nonlinguistic vocalisations using presegmented data. There are very few works on audiovisual laughter detection from unsegmented audiovisual streams and have either been tested on small datasets or use coarse visual features. As a consequence, results are mixed and it is not clear to what extent the addition of visual information to audio is beneficial for laughter detection. In this work, we attempt to overcome the limitation of previous studies and investigate the performance of audiovisual fusion for laughter detection using audiovisual continuous streams from the SEMAINE database. Our results suggest that there is indeed an improvement in laughter detection with the addition of visual information which is dependent on the performance of the voice activity detector.

Automatic Recognition of Affective Laughter in Spontaneous Dyadic Interactions from Audiovisual Signals

“…Experiments on the automatic recognition of laughter from the audiovisual data contained in the SEMAINE-SAL database have been previously reported on a lower amount of instances [35]. Data were partitioned into speaker independent training, validation and test partitions.…”

Perception and Automatic Recognition of Laughter from Whole-Body Motion: Continuous and Categorical Perspectives

“…Work on automatic recognition of laughter has also started to emerge but, as with the synthesis of laughter, has mostly focused on the acoustic modality e.g., [29], [30], [31], [32], [33], [34] and more recently on the combination of face and voice cues [35], [36], [37]. Fukushima et al used electromyographic sensors to measure diaphragmatic activity, which drives laughter vocalisations, to detect laughter in people watching television [38].…”

Social Processes: Self-supervised Meta-learning Over Conversational Groups for Forecasting Nonverbal Social Cues

“…In literature, behavior forecasting works mainly consider data at two representations with an increasing level of abstraction: low-level cues or features that are extracted manually or automatically from raw audiovisual data, and manually labeled high-order events or actions. The forecasting task has primarily been formulated to predict future event or action labels from observed cues or other high-order event or action labels [5,6,[9][10][11][12][13]. Moreover, identifying patterns predictive of certain semantic events has been a long-standing topic of focus in the social sciences, where researchers primarily employ a top-down workflow.…”

“…exploratory or confirmatory analysis [14,15]. Examples of such semantic events include speaker turn transitions [5,6], mimicry episodes [13], the termination of an interaction [9,10], or high-order social actions [11,12].…”