On Assessing Driver Awareness of Situational Criticalities: Multi-modal Bio-Sensing and Vision-Based Analysis, Evaluations, and Insights

Abstract

Automobiles for our roadways are increasingly utilizing advanced driver assistance systems. These changes require us to develop novel perception systems not only for accurately understanding the situations and context of these vehicles, but also to understand the awareness of the driver in differentiating between safe and critical situations. The research presented in this paper focused on this specific problem. Even after the development of wearable and compact multi-modal bio-sensing systems in recent years, their application in driver awareness context has been scarcely explored. The capability of simultaneously recording different kinds of bio-sensing data in addition to traditionally used computer vision systems provide exciting opportunities to explore the limitations of these modalities. In this work, we explore the applications of three different bio-sensing modalities namely electroencephalogram (EEG), photoplethysmogram (PPG) and galvanic skin response (GSR) along with a camera-based vision system in driver awareness context. We assess the information from these sensors independently and together using both signal processing and deep learning tools. We show that our methods outperform previously reported studies in the context of monitoring driver awareness and detecting hazardous/non-hazardous situations for short time scales of two-seconds. We verify our methods by collecting user data on twelve subjects for two real-world driving datasets among which one is publicly available (KITTI dataset) and one that we collect ourselves (LISA dataset) with the vehicle being driven in an autonomous mode. This work presents an exhaustive evaluation of multiple sensor modalities on two different datasets for attention monitoring and hazardous events classification.

Machine learning models and facial regions videos for estimating heart rate: a review on Patents, Datasets and Literature

“…When there is ROI clipping, [42] pointed out that, despite requiring less processing in terms of treating irrelevant regions, the challenge of providing a consistent detection of face landmarks to smooth noisy signals occurred. [44,10,36,42,39,45,11,9,43,18,12,37,40,38,6] estimated heart rate using face videos, while [9,46,5,1,41] estimated additional physiological signals using contact devices or heart rate estimation techniques.…”

“…The strategy of approaching heartbeat estimation as a classification problem in [40] resulted in improved network Machine learning models and facial regions videos for estimating heart rate: a review on Patents, Datasets and Literature performance in estimating heart rate from videos with dynamic circumstances. In contrast, it was discovered in [5] that Heart Rate Variability (HRV) is an excellent metric for classifying cognitive states and is also more robust than heart rate.…”

“…Heart rate estimation is particularly important for monitoring users in a range of frequent situations, such as driving a vehicle [1], engaging in physical activities [2], working in hazardous conditions [3], and during investigative police interviews [4]. Heart rate or its variability may be used to track and detect stress-related aspects [1], tiredness [5], emotions [6], health [7], and social behavior [8].…”

Remote Heart Rate Prediction in Virtual Reality Head-Mounted Displays Using Machine Learning Techniques

“…The estimation of heart rate is essential for monitoring humans in a variety of situations, such as driving a vehicle [ 1 ], practicing physical activities [ 2 ], working under hazardous conditions [ 3 ] or during an investigative police interrogation [ 4 ]. The variability of heart rate can be used to map and identify stress [ 1 ], fatigue [ 5 ], emotions [ 6 ], health [ 7 ] and social behavior [ 8 ] indicators.…”

Recent Progress in Bio‐Integrated Intelligent Sensing System

“…[18][19][20][21][22] First, compared to the single-modal data, the multimodal data can carry more useful information. [23,24] Therefore, the overall recognition accuracy increases with additional module; similarly, the accuracy requirement for each module reduces while keeping the same overall accuracy. For example, chemical sensing can greatly increase the recognition accuracy of human physiological conditions when combined with electrophysiological sensing.…”