Assessing Cognitive Performance Using Physiological and Facial Features

Abstract

Sensing and machine learning advances have enabled the unobtrusive measurement of physiological responses and facial expressions so as to estimate one's cognitive performance. This often boils down to mapping the states of the cognitive processes underpinning human cognition: physiological responses (e.g., heart rate) and facial expressions (e.g., frowning) often reflect the states of our cognitive processes. However, it remains unclear whether physiological responses and facial expressions used in one particular task (e.g., gaming) can reliably assess cognitive performance in another task (e.g., coding), because complex and diverse tasks often require varying levels and combinations of cognitive processes. In this paper, we measure the cross-task reliability of physiological and facial responses. Specifically, we assess cognitive performance based on physiological responses and facial expressions for 123 participants in 4 independent studies (3 studies for out-of-sampling training and testing, and 1 study for evaluation only): (1) a Pac-Man game, (2) an adaptive-assessment task, (3) a code-debugging task, and (4) a gaze-based game. We follow an ensemble learning approach after cross-training and cross-testing with all possible combinations of the 3 first datasets. We save the 4th dataset only for testing purposes, and we showcase how to engineer generalizable features that predict cognitive performance. Our results show that the extracted features do generalize, and can reliably predict cognitive performance across a diverse set of cognitive tasks that require different combinations of problem-solving, decision-making, and learning processes for their completion.

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

“…However, research in ML suggests testing data from various distributions (or different studies [ 23 ]) to obtain a reliable estimate of generalizability [ 12 ]. A few MMLA researchers [ 5 , 17 , 24 ] have taken this into consideration while assessing their models. They have gone a step further by evaluating their model’s generalizability across student groups [ 24 ], datasets [ 17 ] and contexts [ 5 ].…”

“…A few MMLA researchers [ 5 , 17 , 24 ] have taken this into consideration while assessing their models. They have gone a step further by evaluating their model’s generalizability across student groups [ 24 ], datasets [ 17 ] and contexts [ 5 ]. For instance, Sharma et al [ 5 ] investigated their models’ performance across different tasks and contexts using leave-one study out evaluation.…”

“…They have gone a step further by evaluating their model’s generalizability across student groups [ 24 ], datasets [ 17 ] and contexts [ 5 ]. For instance, Sharma et al [ 5 ] investigated their models’ performance across different tasks and contexts using leave-one study out evaluation. Prieto et al [ 17 ] used leave-one session out evaluation to test the generalizability of their model across datasets.…”

“…This desired generalizability can be assessed by evaluating models on context generalizability. Few MMLA researchers have evaluated their models at this level [ 5 , 17 ]. Although it is the most desirable generalizability, greater effort is required to achieve it.…”

“…Multimodal Learning Analytics (MMLA) extends Learning Analytics (LA) by gathering data from digital and physical spaces to gain a holistic picture of the learning process [ 1 , 2 , 3 ]. Besides traditional digital logs, MMLA researchers have used a variety of sensors (e.g., cameras, microphones, eye-trackers) [ 4 , 5 , 6 ] to gather data about learning activities, student behavior, physiology and the learning environment [ 7 ]. Establishing the connection between these data sources and learning can also help to bridge the gap between learning science and machine learning (ML) techniques [ 4 , 7 ].…”