Multi-modal affect detection for learning applications

Gogia, Yash; Singh, Eejya; Mohatta, Shreyash; Sreejith,

doi:10.1109/tencon.2016.7848760

Cited by 4 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, researchers believe that multi-sensor data fusion methodology has ability to increase the accuracy and reliability of the estimates [49], which shows the significance and feasibility of multi-channel data fusion methodology in diverse research fields. Gogia et al [50] used facial features and brain signal of user captured from a camera and a Brain Computer Interfacer (BCI) module. Di Mitri et al [19] proposed an approach based on multimodal data such as heart rate, step count, weather condition and learning activity that can be used to predict learning performance in self-regulated learning settings.…”

Section: B State-of-the-art Methods and Techniques On Detecting Engagementmentioning

confidence: 99%

Recognizing Multidimensional Engagement of E-Learners Based on Multi-Channel Data in E-Learning Environment

et al. 2019

View full text Add to dashboard Cite

Lack of supervision'' is a particularly challenging problem in E-learning or distance learning environments. A wide range of research efforts and technologies have been explored to alleviate its impact by monitoring students' engagement, such as emotion or learning behaviors. However, the current research still lacks multi-dimensional computational measures for analyzing learner's engagement from the interactions that occur in digital learning environment. In this paper, we propose an integrated framework to identify learning engagement from three facets: affect, behavior and cognitive state, which are conveyed by learner's facial expressions, eye movement behaviors and the overall performance during short video learning session. To recognize the three states of learners, three channel data is recorded: 1) video/image sequence captured by camera; 2) eye movement information from a non-intrusive and cost-effective eye tracker; and 3) click stream data from mouse. Based on these modalities, a multi-channel data fusion strategy is designed that concatenates time series features of three channels in the same time segment to predict course learning performance. We also presented a new method to make the self-reported annotations more reliable without using external observers' verification. To validate the approach and methods, 46 participants were invited to attend a representative on-line course that consists of short videos in our designed learning environment. The results demonstrated the effectiveness of the proposed framework and methods in monitoring learning engagement. More importantly, a prototype system was developed to detect learner's emotional and eye behavioral engagement in real-time as well as predict the learning performance of learners after they had completed each short video course.INDEX TERMS E-learning, engagement recognition, multi-channel data fusion, learning performance prediction.

show abstract

Section: B State-of-the-art Methods and Techniques On Detecting Engagementmentioning

confidence: 99%

Recognizing Multidimensional Engagement of E-Learners Based on Multi-Channel Data in E-Learning Environment

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Also, the human manipulation computer completes their relevant instruction and emotion calculations are held by the computer intelligence [7], [8]. In the field of affective computing [9], emotion recognition is the important basic component [10]. In order to create a peaceful and orderly environment for human-machine communication, it must provide the computer system with significant and allencompassing intelligence, as evidenced by the numerous forms of emotional data that have been studied [11].Numerous studies have been conducted to examine human emotions and have discovered that emotional variations lead to changes in behaviour, psychology, and physical health.…”

Section: Introductionmentioning

confidence: 99%

“…S is constructed by promising performances of various features for face representation such that it can extract various texture and local information [31]. The 2D-Gabor filters [9] are specified in eq. ( 4), ( )…”

mentioning

confidence: 99%

Deep Hunting Social Optimization Algorithm Enabled Deep Neuro Fuzzy Network for Facial Expression Recognition

Komala,

Jayadevappa,

2024

Preprint

View full text Add to dashboard Cite

Recently, a number of researchers have been able to identify human emotions from facial expressions. Realistically speaking, emotion communication is a transient and multimodal process, which makes identifying human emotions difficult. This paper proposes an effective model for multimodal facial expression recognition using the Deer Hunting Social Optimization Algorithm (DHSOA) based Deep Neuro Fuzzy Network (DNFN) in order to overcome such problems in the existing models and achieve outstanding performance in the field of human facial expression recognition. In this work, input video and input electroencephalogram (EEG) signals are considered for additional processing. The feature selection stage uses DHSOA to select appropriate features from input video and EEG signal phases. In contrast, the suggested DHSOA-based DNFN for facial video has a maximum accuracy of 0.910, a precision of 0.899, and a recall of 0.900.

show abstract

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Wang

Zhou

Wen-lang

et al. 2020

Int. J. Mach. Learn. & Cyber.

View full text Add to dashboard Cite

Multi-modal affect detection for learning applications

Cited by 4 publications

References 7 publications

Recognizing Multidimensional Engagement of E-Learners Based on Multi-Channel Data in E-Learning Environment

Recognizing Multidimensional Engagement of E-Learners Based on Multi-Channel Data in E-Learning Environment

Deep Hunting Social Optimization Algorithm Enabled Deep Neuro Fuzzy Network for Facial Expression Recognition

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Contact Info

Product

Resources

About