Classification of Driver Cognitive Load: Exploring the Benefits of Fusing Eye-Tracking and Physiological Measures

Abstract

In-vehicle infotainment systems can increase cognitive load and impair driving performance. These effects can be alleviated through interfaces that can assess cognitive load and adapt accordingly. Eye-tracking and physiological measures that are sensitive to cognitive load, such as pupil diameter, gaze dispersion, heart rate (HR), and galvanic skin response (GSR), can enable cognitive load estimation. The advancement in cost-effective and nonintrusive sensors in wearable devices provides an opportunity to enhance driver state detection by fusing eye-tracking and physiological measures. As a preliminary investigation of the added benefits of utilizing physiological data along with eye-tracking data in driver cognitive load detection, this paper explores the performance of several machine learning models in classifying three levels of cognitive load imposed on 33 drivers in a driving simulator study: no external load, lower difficulty 1-back task, and higher difficulty 2-back task. We built five machine learning models, including k-nearest neighbor, support vector machine, feedforward neural network, recurrent neural network, and random forest (RF) on (1) eye-tracking data only, (2) HR and GSR, (3) eye-tracking and HR, (4) eye-tracking and GSR, and (5) eye-tracking, HR, and GSR. Although physiological data provided 1%–15% lower classification accuracies compared with eye-tracking data, adding physiological data to eye-tracking data increased model accuracies, with an RF classifier achieving 97.8% accuracy. GSR led to a larger boost in accuracy (29.3%) over HR (17.9%), with the combination of the two factors boosting accuracy by 34.5%. Overall, utilizing both physiological and eye-tracking measures shows promise for driver state detection applications.

ADABase: A Multimodal Dataset for Cognitive Load Estimation

“…In general, we can observe that combining modalities leads to an improvement in classification accuracy. This is especially evident in the case of using data from both tasks and emphasizes the importance of a multimodal approach, given that other publications report similar results [ 116 , 117 , 118 ]. In addition, we report the gain feature importance values of our XGBoost classifier, showing important contributions of features like heart rate, heart rate variability, change of skin conductance within a given time interval, IPA, fixations and saccades of our eye tracker data as features with strong predictive power in our binary low-vs-high cognitive load classification task.…”

Capturing self‐regulated learning processes in virtual reality: Causal sequencing of multimodal data

“…Cognitive load has been measured using a variety of approaches including self‐report measures (eg, NASA‐TLX; Hart, 2006), heart‐rate monitoring (eg, Fortenbacher et al, 2019), electroencephalography, (EEG; eg, Antonenko et al, 2010), linguistic features (eg, Chen et al, 2016), gestures (eg, Ruiz et al, 2007), Galvanic Skin Response (GSR; eg, Buchwald et al, 2019), and eye‐based signals (eg, He et al, 2022). For example, Fortenbacher et al (2019) developed a sensor‐ based adaptive learning system.…”

Understanding the domain of driving distraction with knowledge graphs

“…As shown in the graph, "cellphone" turns out to be the most concerned issue and is related to entities like "passenger", "eating/smoking/ drinking", "gender difference" and "age difference", which all could be interpreted as classical and significant hotspots of the domain [36][37][38]. Algorithms like "deep learning", "random forest" and "support vector machine" are grouped together, indicating that some studies may lay emphasis on comparison or integration of algorithms in application of the driving distraction domain [39]. More local connections could be identified as well.…”