Toward automating a human behavioral coding system for married couples’ interactions using speech acoustic features

Black, Matthew; Katsamanis, Athanasios; Baucom, Brian R.; Lee, Chi-Chun; Lammert, Adam C.; Christensen, Andrew; Georgiou, Panayiotis G.; Narayanan, Shrikanth

doi:10.1016/j.specom.2011.12.003

Cited by 98 publications

(88 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In comparison to the values of neutral statements, there was a significant difference in the pitch of statements classified as cues and concerns by the VR-CoDES. The statistical difference, higher f0 averages, indicates a trend consistent with the literature that individuals use an increase in the pitch of their voice when speaking with anxiety or other emotional arousal [13][14][15][16][17][18][19]. It is difficult to assign a level that would be considered a clinical difference.…”

Section: Discussionsupporting

confidence: 70%

“…Reviewing the literature on vocal characteristics of human affect, fundamental frequency of pitch (f0), has been identified as one of the most reliable tools, essential for detecting emotional arousal using the voice [13][14][15][16][17][18][19]. However, no study has yet used this objective measure of emotional distress to characterise patient concern in the clinical oncology setting.…”

Section: Fundamental Frequency Of Pitch (F0)mentioning

confidence: 99%

See 1 more Smart Citation

Using fundamental frequency of cancer survivors’ speech to investigate emotional distress in out-patient visits

Kandsberger

Rogers

Zhou

et al. 2016

Patient Education and Counseling

View full text Add to dashboard Cite

2 Highlights  Emotions in clinical conversations of cancer patients studied by a coding system and speech prosody  Emotional energy f0 was associated with cues and concerns as coded by VR-CoDES system  An additional aid to study emotional speech with potential to reveal hidden content and meaning AbstractObjective: Emotions, are in part conveyed by varying levels of fundamental frequency of voice pitch (f0). This study tests the hypothesis that patients display heightened levels of emotional arousal (f0) during Verona Coding Definitions of Emotional Sequences (VR-CoDES) cues and concerns versus during neutral statements. Methods:The audio recordings of sixteen head and neck cancer survivors' follow-up consultations were coded for patients' emotional distress. Pitch (f0) of coded cues and concerns, including neutral statements was extracted. These were compared using a hierarchical linear model, nested for patient and pitch range, controlling for statement speech length. Utterance content was also explored. Results:Clustering by patient explained 30% of the variance in utterances f0. Cues and concerns were on average 13.07 Hz higher than neutral statements (p = 0.02). Cues and concerns in these consultations contained content with a high proportion of recurrence fears. Conclusion:The present study highlights the benefits and challenges of adding f0 and potential other prosodic features to the toolkit of coding emotional distress in the health communication setting. Practice implications:The assessment of f0 during clinical conversations can provide additional information for research into emotional expression.3

show abstract

Section: Discussionsupporting

confidence: 70%

Section: Fundamental Frequency Of Pitch (F0)mentioning

confidence: 99%

Using fundamental frequency of cancer survivors’ speech to investigate emotional distress in out-patient visits

Kandsberger

Rogers

Zhou

et al. 2016

Patient Education and Counseling

View full text Add to dashboard Cite

show abstract

“…Following this, we perform the feature extraction from speech regions. In our work we employ the preprocessing steps described in [12]. In short: We employ all available interactions with a SNR above 5dB, and perform VAD and Diarization.…”

Section: Audio Preprocessingmentioning

confidence: 99%

“…Over the last few years Behavioral Signal Processing (BSP) [9,10] has examined the analysis of such complex, domain specific behaviors. Based on machine learning techniques, BSP employed lexical [11], acoustic [12], and visual [13,14] information to analyze and model multimodal human behaviors. For instance, in couples' therapy domain, Black et al [12] built an automatic human behavioral coding system for couples interaction by using acoustic features.…”

Section: Introductionmentioning

confidence: 99%

Sparsely Connected and Disjointly Trained Deep Neural Networks for Low Resource Behavioral Annotation: Acoustic Classification in Couples’ Therapy

Li¹,

Baucom²,

Georgiou³

2016

Interspeech 2016

View full text Add to dashboard Cite

Observational studies are based on accurate assessment of human state. A behavior recognition system that models interlocutors' state in real-time can significantly aid the mental health domain. However, behavior recognition from speech remains a challenging task since it is difficult to find generalizable and representative features because of noisy and high-dimensional data, especially when data is limited and annotated coarsely and subjectively. Deep Neural Networks (DNN) have shown promise in a wide range of machine learning tasks, but for Behavioral Signal Processing (BSP) tasks their application has been constrained due to limited quantity of data.We propose a Sparsely-Connected and Disjointly-Trained DNN (SD-DNN) framework to deal with limited data. First, we break the acoustic feature set into subsets and train multiple distinct classifiers. Then, the hidden layers of these classifiers become parts of a deeper network that integrates all feature streams. The overall system allows for full connectivity while limiting the number of parameters trained at any time and allows convergence possible with even limited data. We present results on multiple behavior codes in the couples' therapy domain and demonstrate the benefits in behavior classification accuracy. We also show the viability of this system towards live behavior annotations.

show abstract

“…In particular, human speech contains rich information for effectively conveying emotions and communicating wants, needs, and desires. The richness of human speech for understanding emotions within human interactions has motivated researchers to explore the area of emotion classification based on speech Black et al (2013).…”

Section: Introductionmentioning

confidence: 99%

Enhanced multiclass SVM with thresholding fusion for speech-based emotion classification

Yang

Yuan

Zhou

et al. 2016

Int J Speech Technol

View full text Add to dashboard Cite

As an essential approach to understanding human interactions, emotion classification is a vital component of behavioral studies as well as being important in the design of context-aware systems. Recent studies have shown that speech contains rich information about emotion, and numerous speech-based emotion classification methods have been proposed. However, the classification performance is still short of what is desired for the algorithms to be used in real systems. We present an emotion classification system using several one-againstall support vector machines with a thresholding fusion mechanism to combine the individual outputs, which provides the functionality to effectively increase the emotion classification accuracy at the expense of rejecting some samples as unclassified. Results show that the proposed system outperforms three stateof-the-art methods and that the thresholding fusion mechanism can effectively improve the emotion classification, which is important for applications that require very high accuracy but do not require that all samples be classified.We evaluate the system performance for several challenging scenarios including speaker-independent tests, tests on noisy speech signals, and tests using non-professional acted recordings, in order to demonstrate the performance of the system and the effectiveness of the thresholding fusion mechanism in real scenarios.

show abstract

Toward automating a human behavioral coding system for married couples’ interactions using speech acoustic features

Cited by 98 publications

References 61 publications

Using fundamental frequency of cancer survivors’ speech to investigate emotional distress in out-patient visits

Using fundamental frequency of cancer survivors’ speech to investigate emotional distress in out-patient visits

Sparsely Connected and Disjointly Trained Deep Neural Networks for Low Resource Behavioral Annotation: Acoustic Classification in Couples’ Therapy

Enhanced multiclass SVM with thresholding fusion for speech-based emotion classification

Contact Info

Product

Resources

About