Detection of Laughter-in-Interaction in Multichannel Close-Talk Microphone Recordings of Meetings

Abstract

Abstract. Laughter is a key element of human-human interaction, occurring surprisingly frequently in multi-party conversation. In meetings, laughter accounts for almost 10% of vocalization effort by time, and is known to be relevant for topic segmentation and the automatic characterization of affect. We present a system for the detection of laughter, and its attribution to specific participants, which relies on simultaneously decoding the vocal activity of all participants given multi-channel recordings. The proposed framework allows us to disambiguate laughter and speech not only acoustically, but also by constraining the number of simultaneous speakers and the number of simultaneous laughers independently, since participants tend to take turns speaking but laugh together.

Audiovisual Discrimination Between Speech and Laughter: Why and When Visual Information Might Help

“…In the first group, there are usually two approaches: 1) laughter detection/segmentation, e.g., [27], [29], [31], where the aim is to segment an unsegmented audio stream into laughter and nonlaughter episodes; 2) laughter-versus-speech classification/discrimination, e.g., [33], [57], [60], where the aim is to correctly classify presegmented episodes of laughter and speech. One of the first works on laughter detection is that of Kennedy and Ellis [27], who trained SVMs with MFCCs, spatial cues, and modulation spectrum features (MSFs) to detect group laughter, i.e., when more than a certain percentage of participants are laughing.…”

“…GMMs were trained for speech, laughter, and silence, and the system was evaluated on the ICSI corpus achieving an EER of 10.9%. Laskowski and Schultz [31] present a system for the detection of laughter and its attribution to specific participants in multi-channel recordings. Each participant can be in one of the three states (silence, speech, laughter) and the aim is to decode the vocal activity of all participants simultaneously.…”

How does real affect affect affect recognition in speech?

“…Knox and Mirghafori [94], Knox et al [95] used neural networks and HMMs in combination with MFCCs and prosodic features for laughter segmentation. Laskowski and Schultz [104] used a multiparticipant 3-state vocal activity recognition module to detect so-called laughter-in-interaction. Recently, audiovisual approaches to laughter detection have been undertaken by Reuderink et al [147] and Petridis and Pantic [130,131,132]: according to their work, fusion between the visual and auditory modalities helps, but it remains unclear how this fusion between visual and auditory information should work.…”

“…However, note that they only used annotated vocalized segments; silence and other sounds were thus discarded. Laskowski and Schultz [104] segmented laughter based on three distinct states, namely laughter, speech and non-vocalizations. Their system is not only based on the acoustic characteristics of laughter, but also makes use of the vocal activity of multiple participants by constraining the number of simultaneous speakers and the number of simultaneous laughter.…”

“…Grimm et al [69] and Grimm et al [68] report mean absolute errors between 0.13 and 0.28, and Pearson's ρ correlation coefficients of 0.75-0.82 and 0.28-0.46 for Arousal and Valence prediction respectively. Laskowski and Schultz [104] report human inter-rater agreement κ between 0.15 and 0.67 for 3 human annotators, and Douglas-Cowie et al [54] report human inter-rater agreement κ between 0.37 and 0.54, based on category emotion annotation. Although real valid comparisons are not possible, e.g., due to different databases used, different annotation procedures used etc., a comparison with our results can give us a rough idea of how our results relate to other studies.…”