The Automatic Detection of Cognition Using EEG and Facial Expressions

Abstract

Detecting cognitive profiles is critical to efficient adaptive learning systems that automatically adjust the content delivered depending on the learner’s cognitive states and skills. This study explores electroencephalography (EEG) and facial expressions as physiological monitoring tools to build models that detect two cognitive states, namely, engagement and instantaneous attention, and three cognitive skills, namely, focused attention, planning, and shifting. First, while wearing a 14-channel EEG Headset and being videotaped, data has been collected from 127 subjects taking two scientifically validated cognitive assessments. Second, labeling was performed based on the scores obtained from the used tools. Third, different shallow and deep models were experimented in the two modalities of EEG and facial expressions. Finally, the best performing models for the analyzed states are determined. According to the used performance measure, which is the f-beta score with beta = 2, the best obtained results for engagement, instantaneous attention, and focused attention are EEG-based models with 0.86, 0.82, and 0.63 scores, respectively. As for planning and shifting, the best performing models are facial expressions-based models with 0.78 and 0.81, respectively. The obtained results show that EEG and facial expressions contain important and different cues and features about the analyzed cognitive states, and hence, can be used to automatically and non-intrusively detect them.

Wearable Biosensor Technology in Education: A Systematic Review

“…EEG-based technologies can also be used as predictors of cognitive performance [ 28 ] by using the alpha/theta ratio and delta band power (which are indicators of mental fatigue and drowsiness). Alongside facial expressions, EEG can be predictive of states of engagement, attention [ 113 ], planning [ 114 ], shifting [ 115 ], and even student effort [ 116 ]. Regarding attention, considering that it is the most important factor in learning, protocols have been proposed to classify the levels of attention in educational environments [ 117 ].…”

“…By harnessing these technologies in educational settings, it is possible to unlock endless possibilities for personalized and immersive learning experiences [ 28 , 34 , 35 , 102 , 130 ]. The exponential advances in this field have developed new ways to improve education, but with this growth comes several challenges that must be addressed to ensure improved learning outcomes [ 115 ].…”

Wearable Biosensor Technology in Education: A Systematic Review

“…EEG-based technologies can also be used as predictors of cognitive performance [29] by using alpha/theta ratio and delta band power (which are indicators of mental fatigue and drowsiness). Alongside facial expressions, EEG can be predictive of states of engagement, attention [118], planning [119], shifting [120] and even student effort [121]. Regarding attention, considering that it is the most important factor in learning, protocols have been proposed to classify the levels of attention in educational environments [122].…”

Three-dimensional DenseNet self-attention neural network for automatic detection of student’s engagement

“…Although their system aims to improve learning by analyzing instructor-learner interactions in the classroom, it does not reflect the level of engagement of learners. Kerdawy et al [ 52 ] proposed an approach to use electroencephalography (EEG) and facial expression modalities to anticipate students’ cognitive states, engagement, and spontaneous attention. They observed that, while the EEG and face-based models demonstrated significant agreement in the engaged classes, there was less consensus in the non-engaged case.…”