Audiovisual Discrimination Between Speech and Laughter: Why and When Visual Information Might Help

Petridis, Stavros; Pantić, Maja

doi:10.1109/tmm.2010.2101586

Cited by 43 publications

(48 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Inspired by [51], we follow a face-anatomy-driven rather than a simply data-driven approach to identifying the most suitable feature representation of facial shape for the problem at hand. To this end, we visually inspect the deformation pattern associated with each component of the ASM.…”

Section: Featuresmentioning

confidence: 99%

The Conflict Escalation Resolution (CONFER) Database

Georgakis

Panagakis

Zafeiriou

et al. 2017

Image and Vision Computing

Self Cite

View full text Add to dashboard Cite

Conflict is usually defined as a high level of disagreement taking place when individuals act on incompatible goals, interests, or intentions. Research in human sciences has recognized conflict as one of the main dimensions along which an interaction is perceived and assessed. Hence, automatic estimation of conflict intensity in naturalistic conversations would be a valuable tool for the advancement of human-centered computing and the deployment of novel applications for social skills enhancement including conflict management and negotiation. However, machine analysis of conflict is still limited to just a few works, partially due to an overall lack of suitable annotated data, while it has been mostly approached as a conflict or (dis)agreement detection problem based on audio features only. In this work, we aim to overcome the aforementioned limitations by a) presenting the Conflict Escalation Resolution (CONFER) Database, a set of excerpts from audio-visual recordings of televised political debates where conflicts naturally arise, and b) reporting baseline experiments on audio-visual conflict intensity estimation. The database contains approximately 142 minutes of recordings in Greek language, split over 120 non-overlapping episodes of naturalistic conversations that involve two or three interactants. Subject-and session-independent experiments are conducted on continuous-time (frame-by-frame) estimation of real-valued conflict intensity, as opposed to binary conflict/non-conflict clas- * Corresponding author. E-mail address: christos.georgakis@imperial.ac.uk Preprint submitted to Image and Vision Computing December 20, 2016 A C C E P T E D M A N U S C R I P T ACCEPTED MANUSCRIPTsification. For the problem at hand, the efficiency of various audio and visual features and fusion of them as well as various regression frameworks is examined. Experimental results suggest that there is much room for improvement in the design and development of automated multi-modal approaches to continuous conflict analysis. The CONFER Database is publicly available for non-commercial use at http://ibug.doc.ic.ac.uk/resources/confer/.

show abstract

Section: Featuresmentioning

confidence: 99%

The Conflict Escalation Resolution (CONFER) Database

Georgakis

Panagakis

Zafeiriou

et al. 2017

Image and Vision Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…We should also point out that the performance of laughter detection is far from the performance of laughtervs-speech classification based on presegmented episodes where F1 measures close to 90% are achieved [13], [15]. It is also difficult to compare with other approaches, given the completely different datasets and features used.…”

Section: Resultsmentioning

confidence: 96%

“…Cepstral features, such as MFCCs, have been widely used in speech recognition and have also been successfully used for laughter detection [2], [3] and laughter-vs-speech discrimination [13], [15]. Therefore, we use 13 MFCCs in this study as well, which are computed every 10ms over a window of 40ms, i.e.…”

Section: A Audio Featuresmentioning

confidence: 99%

“…Only recently, few works on audiovisual discrimination between speech and laughter [13], [14], [15], [16], [17] or between laughter, hesitation and consent [18] based on presegmented episodes have been published. However, work on audiovisual detection of laughter or other nonlinguistic vocalisations using continuous audiovisual streams is limited.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Audiovisual Detection of Laughter in Human-Machine Interaction

Petridis

Leveque

Pantić

2013

2013 Humaine Association Conference on Affective Computing and Intelligent Interaction

Self Cite

View full text Add to dashboard Cite

Abstract-Laughter is clearly an audiovisual event, consisting of the laughter vocalization and of facial activity, mainly around the mouth and sometimes in the upper face. However, past research on laughter recognition has mainly focused on the information available in the audio channel only, mainly due to the lack of suitable audiovisual data. Only recently few works have been published which combine audio and visual information and most of them deal with the problem of discriminating laughter from speech or other nonlinguistic vocalisations using presegmented data. There are very few works on audiovisual laughter detection from unsegmented audiovisual streams and have either been tested on small datasets or use coarse visual features. As a consequence, results are mixed and it is not clear to what extent the addition of visual information to audio is beneficial for laughter detection. In this work, we attempt to overcome the limitation of previous studies and investigate the performance of audiovisual fusion for laughter detection using audiovisual continuous streams from the SEMAINE database. Our results suggest that there is indeed an improvement in laughter detection with the addition of visual information which is dependent on the performance of the voice activity detector.

show abstract

“…Speech can be considered as an indirect biosignal that is very well suited to unveil the emotional state of a person. Non-speech utterances have also been shown to be of interest for emotion-aware computing [78]; however, they are relatively unexplored. The audio recordings used for speech processing suffer from various types of noise.…”

Section: Ubiquitous Signals Of Emotionmentioning

confidence: 99%

Ubiquitous emotion-aware computing

Broek

2011

Pers Ubiquit Comput

View full text Add to dashboard Cite

Emotions are a crucial element for personal and ubiquitous computing. What to sense and how to sense it, however, remain a challenge. This study explores the rare combination of speech, electrocardiogram, and a revised Self-Assessment Mannequin to assess people's emotions. 40 people watched 30 International Affective Picture System pictures in either an office or a living-room environment. Additionally, their personality traits neuroticism and extroversion and demographic information (i.e., gender, nationality, and level of education) were recorded. The resulting data were analyzed using both basic emotion categories and the valence-arousal model, which enabled a comparison between both representations. The combination of heart rate variability and three speech measures (i.e., variability of the fundamental frequency of pitch (F0), intensity, and energy) explained 90% (p \ .001) of the participants' experienced valence-arousal, with 88% for valence and 99% for arousal (ps \ .001). The six basic emotions could also be discriminated (p \ .001), although the explained variance was much lower: 18-20%. Environment (or context), the personality trait neuroticism, and gender proved to be useful when a nuanced assessment of people's emotions was needed. Taken together, this study provides a significant leap toward robust, generic, and ubiquitous emotion-aware computing.

show abstract

Audiovisual Discrimination Between Speech and Laughter: Why and When Visual Information Might Help

Cited by 43 publications

References 47 publications

The Conflict Escalation Resolution (CONFER) Database

The Conflict Escalation Resolution (CONFER) Database

Audiovisual Detection of Laughter in Human-Machine Interaction

Ubiquitous emotion-aware computing

Contact Info

Product

Resources

About