Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice

Mitra, Vikramjit; Booker, Sue; Marchi, Erik; Farrar, David Scott; Peitz, Ute Dorothea; Cheng, Bridget; Teves, Ermine A.; Mehta, Anuj; Naik, Devang

doi:10.48550/arxiv.1907.00112

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Emotion classification is also beneficial in the para-linguistic field as well. Recently, the commercial digital assistant application, such as Siri, found that para-linguistic information, such as emotion, is beneficial for recognizing the intent of the speaker [2]. Humans usually employ multi-modality information to identify emotions.…”

Section: Introductionmentioning

confidence: 99%

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Yoon

Dey

Lee

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this work, we explore the impact of visual modality in addition to speech and text for improving the accuracy of the emotion detection system. The traditional approaches tackle this task by fusing the knowledge from the various modalities independently for performing emotion classification. In contrast to these approaches, we tackle the problem by introducing an attention mechanism to combine the information. In this regard, we first apply a neural network to obtain hidden representations of the modalities. Then, the attention mechanism is defined to select and aggregate important parts of the video data by conditioning on the audio and text data. Furthermore, the attention mechanism is again applied to attend important parts of the speech and textual data, by considering other modality. Experiments are performed on the standard IEMOCAP dataset using all three modalities (audio, text, and video). The achieved results show a significant improvement of 3.65% in terms of weighted accuracy compared to the baseline system.

show abstract

Section: Introductionmentioning

confidence: 99%

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Yoon

Dey

Lee

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…For uni-and multi-modal inputs, 2-layer TC-GRU models were trained, where the performance on a held-out Valid1.6 set (see Table 1) was used for model selection. Concordance correlation coefficient (CCC) [21] is used as the loss function (L ccc ) (see (1)), where L ccc is a combination (α = 1/3 and β = 1/3) of CCC's obtained from each of the three dimensional emotions. CCC is defined by (2), where µ x and µ y are the means, σ 2 x and σ 2 y are the corresponding variances for the estimated and ground-truth variables, and ρ is the correlation coefficient between those variables.…”

Section: Model Trainingmentioning

confidence: 99%

“…Current humanmachine interaction systems can recognize the words said by the speaker but fail to acknowledge the expressed emotion. Reliable and robust speech-based emotion models can help improve human-computer interaction and health/wellness applications, such as voice assistants [1,2], clinical mental health diagnoses [3,4], and/or therapy treatments [5].…”

Section: Introductionmentioning

confidence: 99%

Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis

Mitra¹,

Kowtha²,

Chien³

et al. 2023

Preprint

View full text Add to dashboard Cite

Pre-trained model representations have demonstrated stateof-the-art performance in speech recognition, natural language processing, and other applications. Speech models, such as Bidirectional Encoder Representations from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled generating lexical and acoustic representations to benefit speech recognition applications. We investigated the use of pre-trained model representations for estimating dimensional emotions, such as activation, valence, and dominance, from speech. We observed that while valence may rely heavily on lexical representations, activation and dominance rely mostly on acoustic information. In this work, we used multi-modal fusion representations from pre-trained models to generate state-of-the-art speech emotion estimation, and we showed a 100% and 30% relative improvement in concordance correlation coefficient (CCC) on valence estimation compared to standard acoustic and lexical baselines. Finally, we investigated the robustness of pre-trained model representations against noise and reverberation degradation and noticed that lexical and acoustic representations are impacted differently. We discovered that lexical representations are more robust to distortions compared to acoustic representations, and demonstrated that knowledge distillation from a multi-modal model helps to improve the noise-robustness of acoustic-based models.

show abstract

“…In speech communication people make use of two types of information to convey their intentions: what is said (linguistic information) and how it is said (paralinguistic information) [1]. Paralinguistic information is expressed by suprasegmental features such as duration, intensity and pitch.…”

Section: Introductionmentioning

confidence: 99%

“…In the literature of a semantic framework called Alternative Semantics [3,4], focus is understood as the indication of 'the presence of alternatives that are relevant for the interpretation of linguistic expressions' [5]. Consider this example: (1) a. John bought the apple. b. John bought the apple.…”

Section: Introductionmentioning

confidence: 99%

Representing `how you say' with `what you say': English corpus of focused speech and text reflecting corresponding implications

Suzuki¹,

Nakamura²

2022

Preprint

View full text Add to dashboard Cite

In speech communication, how something is said (paralinguistic information) is as crucial as what is said (linguistic information). As a type of paralinguistic information, English speech uses sentence stress, the heaviest prominence within a sentence, to convey emphasis. While different placements of sentence stress communicate different emphatic implications, current speech translation systems return the same translations if the utterances are linguistically identical, losing paralinguistic information. Concentrating on focus, a type of emphasis, we propose mapping paralinguistic information into the linguistic domain within the source language using lexical and grammatical devices. This method enables us to translate the paraphrased text representations instead of the transcription of the original speech and obtain translations that preserve paralinguistic information. As a first step, we present the collection of an English corpus containing speech that differed in the placement of focus along with the corresponding text, which was designed to reflect the implied meaning of the speech. Also, analyses of our corpus demonstrated that mapping of focus from the paralinguistic domain into the linguistic domain involved various lexical and grammatical methods. The data and insights from our analysis will further advance research into paralinguistic translation. The corpus will be published via LDC.

show abstract

Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice

Cited by 3 publications

References 0 publications

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis

Representing `how you say' with `what you say': English corpus of focused speech and text reflecting corresponding implications

Contact Info

Product

Resources

About