Uncertainty-Aware Boosted Ensembling in Multi-Modal Settings

Sarawgi, Utkarsh; Khincha, Rishab; Zulfikar, Wazeer; Ghosh, Satrajit; Maes, Pattie

doi:10.1109/ijcnn52387.2021.9534161

Cited by 2 publications

(1 citation statement)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Tian et al [49] explored the use of uncertainty estimation in fusing the softmax scores predicted using CNNs for semantic segmentation. Other notable approaches to uncertainty-aware multimodal fusion are based on optimal transport for cross-modal correspondence [50], random prior functions [51], boosted ensembles [52], and factorised deep markov models [53].…”

Section: Related Workmentioning

confidence: 99%

COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition

Tellamekala,

Amiriparian,

Schuller

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Automatically recognising apparent emotions from face and voice is hard, in part because of various sources of uncertainty, including in the input data and the labels used in a machine learning framework. This paper introduces an uncertainty-aware multimodal fusion approach that quantifies modality-wise aleatoric or data uncertainty towards emotion prediction. We propose a novel fusion framework, in which latent distributions over unimodal temporal context are learned by constraining their variance. These variance constraints, Calibration and Ordinal Ranking, are designed such that the variance estimated for a modality can represent how informative the temporal context of that modality is w.r.t. emotion recognition. When well-calibrated, modality-wise uncertainty scores indicate how much their corresponding predictions are likely to differ from the ground truth labels. Well-ranked uncertainty scores allow the ordinal ranking of different frames across different modalities. To jointly impose both these constraints, we propose a softmax distributional matching loss. Our evaluation on AVEC 2019 CES, CMU-MOSEI, and IEMOCAP datasets shows that the proposed multimodal fusion method not only improves the generalisation performance of emotion recognition models and their predictive uncertainty estimates, but also makes the models robust to novel noise patterns encountered at test time.

show abstract