Multimodal emotion recognition using SDA-LDA algorithm in video clips

Tiwari, Pradeep; Rathod, Harshil; Thakkar, Sakshee; Darji, Anand D.

doi:10.1007/s12652-021-03529-7

Cited by 9 publications

(9 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is remarkable that our approach surpasses several supervised competitors: [42], [48], [44], [28] with a margin of 2-35% despite working in a more difficult (unsupervised) setting. It also performs on par with supervised approaches: [27], [52]. The results for CMU-MOSEI [43] are given in Table IV.…”

Section: Comparisons With the State-of-the-art Methodsmentioning

confidence: 99%

“…Indeed, it is very common in the MER litreature to apply the feature extraction step separately. This is performed on each modality by using either hand-crafted formulations [52], [53], [29], [26], [27], [43]) and/or deep learning architectures [42], [26], [27]. As example of acoustic features; Log-Mel spectrogram [27], pitch, voiced/unvoiced segmenting features [26], [43], [29], MFCCs [28], [26], [43], [29], features extracted from SoundNet [42]) can be given.…”

Section: Related Workmentioning

confidence: 99%

“…Among the aforementioned approaches, [45], [55] use text and audio, [53], [48], [42], [27], [28], [50], [52] use video and audio, and all others use text, audio and video together. It is worth noting that these techniques are all supervised.…”

Section: Related Workmentioning

confidence: 99%

“…RAVDESS is classbalanced except the neutral class, which was elicited 50% less time than the other emotion classes. We adapted two crossvalidation settings following the methods [42], [48], [27], [28], [13], [72], [44], [53], [12], [52]. The first setting considers the identities of the actors such that the training (validation) and the corresponding testing k-folds have no overlap in terms of actors (shown as actor-split= hereafter).…”

Section: A Datasets and Evaluation Metricsmentioning

confidence: 99%

“…The second setting, instead, applies standard k-fold cross-validation (i.e., actor-split=). In both settings, k was taken as 10 and the reported results are in terms of accuracy (ACC), which is averaged over the 10-folds, supplying fair comparisons with the MER SOTA [42], [48], [27], [28], [44], [53], [52]. As the same statements are being repeated by the actors in RAVDESS dataset [71], the proposed method (as well as the SOTA) are based only on visual and acoustic modalities.…”

Section: A Datasets and Evaluation Metricsmentioning

confidence: 99%

See 4 more Smart Citations

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Franceschini¹,

Fini²,

Beyan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Emotion recognition is involved in several real-world applications. With an increase in available modalities, automatic understanding of emotions is being performed more accurately. The success in Multimodal Emotion Recognition (MER), primarily relies on the supervised learning paradigm. However, data annotation is expensive, time-consuming, and as emotion expression and perception depends on several factors (e.g., age, gender, culture) obtaining labels with a high reliability is hard. Motivated by these, we focus on unsupervised feature learning for MER. We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature. Our end-to-end feature learning approach has several differences (and advantages) compared to existing MER methods: i) it is unsupervised, so the learning is lack of data labelling cost; ii) it does not require data spatial augmentation, modality alignment, large number of batch size or epochs; iii) it applies data fusion only at inference; and iv) it does not require backbones pre-trained on emotion recognition task. The experiments on benchmark datasets show that our method outperforms several baseline approaches and unsupervised learning methods applied in MER. Particularly, it even surpasses a few supervised MER state-of-the-art.

show abstract