Towards multimodal emotion recognition in e-learning environments

Abstract

This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner's facial expressions and verbalizations. FILTWAM's facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants' recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners' behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner's emotional states.

Communication skills training exploiting multimodal emotion recognition

“…While using the judgement of human raters as a reference, the accuracies of our emotion recognition software turned out to be 70% for facial emotions and 61.5% for vocal emotions. Although these accuracies are lower than those of the human raters involved (90% for facial emotions and 89% for vocal emotions), the result is consistent with previous studies (Busso, Deng, & Yildirim, 2004;Jaimes, & Sebe, 2007;Bahreini et al, 2014a;Bahreini et al, 2015a). One may wonder if the FILTWAM emotion recognition software will work better for some emotions than for other ones.…”

“…Further improvements are required to extend the FILTWAM framework for more reliable and mature exploitation of real-time emotion recognition technologies in e learning. This would offer an innovative approach for applying emotion recognition in affective e-learning (Bahreini et al, 2014a;Sebe, 2009). New software applications and serious games with emotion recognition technology could strongly influence e-learning and gaming.…”

“…Other studies have shown that it is possible to measure facial and vocal emotions with considerable reliability in real time, both separately and in combination (Bahreini et al, 2014a;Bahreini et al, 2015a;Bahreini et al, 2015b). Taking the previous research into account, our approach will use common low-cost computer webcams and microphones rather than dedicated sensor systems for emotion detection.…”

Real Time Facial Expression Recognition Using Webcam and SDK Affectiva

“…Drawing upon various published works [4], [10], [31], [2], [30], main system elements from various similar system have been identified. The main elements of each system to evaluate facial expression are:…”

Evaluating the Emotional State of a User Using a Webcam

“…Such devices firstly offer opportunities for more natural interactions with the e-learning applications [1]. Secondly, they offer better ways for gathering affective user data, as they do not interfere with the learning like questionnaires often do.…”