Identifying salient sub-utterance emotion dynamics using flexible units and estimates of affective flow

Provost, Emily Mower

doi:10.1109/icassp.2013.6638345

Cited by 37 publications

(17 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The unweighted recall associated with the PS/Phon and PS/TRA schemes are not statistically significantly different (p<0.05). The importance of variable-length segmentation was also demonstrated in previous work [22]. However, this is the first demonstration of the efficacy of unsupervised variable-length segmentation for raw facial movements.…”

Section: Variable-length Segmentationmentioning

confidence: 53%

Say Cheese vs. Smile

Kim

Provost

2014

Proceedings of the 22nd ACM International Conference on Multimedia

View full text Add to dashboard Cite

Facial movement is modulated both by emotion and speech articulation. Facial emotion recognition systems aim to discriminate between emotions, while reducing the speechrelated variability in facial cues. This aim is often achieved using two key features: (1) phoneme segmentation: facial cues are temporally divided into units with a single phoneme and (2) phoneme-specific classification: systems learn patterns associated with groups of visually similar phonemes (visemes), e.g. /P/, /B/, and /M/. In this work, we empirically compare the effects of different temporal segmentation and classification schemes for facial emotion recognition. We propose an unsupervised segmentation method that does not necessitate costly phonetic transcripts. We show that the proposed method bridges the accuracy gap between a traditional sliding window method and phoneme segmentation, achieving a statistically significant performance gain. We also demonstrate that the segments derived from the proposed unsupervised and phoneme segmentation strategies are similar to each other. This paper provides new insight into unsupervised facial motion segmentation and the impact of speech variability on emotion classification.

show abstract

Section: Variable-length Segmentationmentioning

confidence: 53%

Say Cheese vs. Smile

Kim

Provost

2014

Proceedings of the 22nd ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…The selection of suitable time for the audio segment is a challenging problem in this era. Many researchers have worked, how to select a suitable time for each speech segment which has found some reasonable solution, that a segment of a speech signal is longer than 260ms that have more information to recognize the emotions in his/her speech [33], [34]. In this paper, we have done different observations on multiple frame durations to optimally select 500ms window size to convert single utterance into several segments.…”

Section: A Pre-processing and Sequence Selectionmentioning

confidence: 99%

Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM

2020

View full text Add to dashboard Cite

Emotional state recognition of a speaker is a difficult task for machine learning algorithms which plays an important role in the field of speech emotion recognition (SER). SER plays a significant role in many real-time applications such as human behavior assessment, human-robot interaction, virtual reality, and emergency centers to analyze the emotional state of speakers. Previous research in this field is mostly focused on handcrafted features and traditional convolutional neural network (CNN) models used to extract high-level features from speech spectrograms to increase the recognition accuracy and overall model cost complexity. In contrast, we introduce a novel framework for SER using a key sequence segment selection based on redial based function network (RBFN) similarity measurement in clusters. The selected sequence is converted into a spectrogram by applying the STFT algorithm and passed into the CNN model to extract the discriminative and salient features from the speech spectrogram. Furthermore, we normalize the CNN features to ensure precise recognition performance and feed them to the deep bi-directional long short-term memory (BiLSTM) to learn the temporal information for recognizing the final state of emotion. In the proposed technique, we process the key segments instead of the whole utterance to reduce the computational complexity of the overall model and normalize the CNN features before their actual processing, so that it can easily recognize the Spatio-temporal information. The proposed system is evaluated over different standard dataset including IEMOCAP, EMO-DB, and RAVDESS to improve the recognition accuracy and reduce the processing time of the model, respectively. The robustness and effectiveness of the suggested SER model is proved from the experimentations when compared to state-of-the-art SER methods with an achieve up to 72.25%, 85.57%, and 77.02% accuracy over IEMOCAP, EMO-DB, and RAVDESS dataset, respectively. INDEX TERMS Speech emotion recognition, deep bidirectional long shot term memory, key segment sequence selection, normalization of CNN features, radial-based function network (RBFN).

show abstract

“…[11] Hence, the lack of IS purity in natural expressions of IS should be considered when designing systems to recognize IS of non-stereotypical speech. In emotion profile theory [17], [18], [19], [20], [21], [22], [23], emotion profile recently has been demonstrated that it can interpret the emotion content of ambiguous utterances (i.e., containing shades of multiple affective classes). Inspired by the emotion profiling technique, the IS profile representation is proposed in this study to better interpret the ISs of ambiguous utterances and increase the discriminative abilities for IS recognition in non-stereotypical speech.…”

Section: Is Profile Representation For Is Recognitionmentioning

confidence: 99%

Interaction Style Recognition Based on Multi-Layer Multi-View Profile Representation

Wei

Lin

2017

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Interaction Style (IS) refers to patterns of interaction containing highly contextual and innate information. Awareness of our IS can help us discover interpersonal conflicts and guide us how to interact with others. Recently, automatic IS recognition is becoming increasingly important in the design of a dialogue system for harmonious interaction. With the goal to select appropriate responses, four IS types proposed by Berens are selected as the basis for our study. In this study, multiple views (multi-views) of the utterances during interaction, including emotions and dialogue topics, are recognized first. Inspired by the emotion profile theory, the IS profiles are then extracted using the multi-view features to better characterize the IS of the interactional utterances. Similar to the multilayer architectures in deep neural networks, a multi-layer multi-view IS profile representation method, structured layer by layer through embedding the multi-views, is proposed to better interpret intermediate representations in the feature space of the interactional utterances based on a probabilistic fusion model. The IS is finally recognized by using the Support Vector Machine (SVM) based on the obtained IS profiles. Experimental results demonstrate that the proposed method achieved an encouraging IS recognition accuracy and outperformed the previous method.

show abstract

Identifying salient sub-utterance emotion dynamics using flexible units and estimates of affective flow

Cited by 37 publications

References 17 publications

Say Cheese vs. Smile

Say Cheese vs. Smile

Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM

Interaction Style Recognition Based on Multi-Layer Multi-View Profile Representation

Contact Info

Product

Resources

About