Analysis of different affective state multimodal recognition approaches with missing data-oriented to virtual learning environments

Abstract

In this work, the affective state of users in virtual learning environments is assessed/recognized in terms of continuous arousal and valence dimensions, making use of multimodal information (audio, text and video), whenever any of these modalities are available. In general, virtual learning environments where these three modalities are all the time, are not common; at some moments only the video modality is available, while in others only text or/and video and/or audio. Different approaches using feature-level fusion and decision-level fusion are proposed for multimodal recognition with missing data. Recognizing according to available modalities is studied following the ideas of dropout from neural networks and of variable input length from recurrent neural networks. This proposal is innovative because it represents emotions in the continuous space, which is not common in virtual education; and makes use of the available modalities in a virtual environment in a given moment, which is very common in virtual learning environments because the people are not speaking or writing all the time.

Research on the Filtering and Classification Method of Interactive Music Education Resources Based on Neural Network

“…DL method can effectively overcome the defects of traditional music classification models, thus providing a new opportunity to solve this problem. The DL model enables the computer to learn to correctly recognize songs without relying on rich acoustic and music theory knowledge [ 22 , 23 ]. Accordingly, a residual gated convolution structure RGLU-SE is introduced for music feature extraction by combining with channel AM.…”