DAMA: A Dynamic Classification of Multimodal Ambiguities

Abstract

Ambiguities represent uncertainty but also a fundamental item of discussion for who is interested in the interpretation of languages and it is actually functional for communicative purposes both in human-human communication and in human-machine interaction. This paper faces the need to address ambiguity issues in human-machine interaction. It deals with the identification of the meaningful features of multimodal ambiguities and proposes a dynamic classification method that characterizes them by learning, and progressively adapting with the evolution of the interaction language, by refining the existing classes, or by identifying new ones. A new class of ambiguities can be added by identifying and validating the meaningful features that characterize and distinguish it compared to the existing ones. The experimental results demonstrate improvement in the classification rate over considering new ambiguity classes.

Emotion Classification from Speech and Text in Videos Using a Multimodal Approach

“…When the modal features have been modelled as the joint features vector, emotion classification essentially becomes a classification process. Based on the findings of the previously analyzed studies and the works in [10,[106][107][108], HMMs appear to be appropriate to extract information from multimodal data, and hence for extracting emotions from multimodal data formalized using a multimodal language. An HMM allows us to classify the language sequence data since this approach can be applied in an analogous way to classify text sequence data and proteins [109].…”

“…The authors apply a strict left-to-right model in which each state is transferred to the next. The authors use HMMs as they can represent the differences in the whole structure of multimodal sentences, manage multimodal features, and incorporate temporal frequent pattern analysis for baseball event classification, as set out in [108]. This method was also selected due to its proven effectiveness in the extraction and classification process [107].…”

MuBeFE: Multimodal Behavioural Features Extraction Method

“…The method is used because of its proven effectiveness in extraction and classification process [Grifoni, 2020a]. Besides, as stated in [Grifoni, 2020b], HMMs can represent differences in the whole structure of multimodal sentences managing multimodal features and incorporating temporal frequent pattern analysis for baseball event classification.…”