Looking at the Driver/Rider in Autonomous Vehicles to Predict Take-Over Readiness

Abstract

Continuous estimation the driver's take-over readiness is critical for safe and timely transfer of control during the failure modes of autonomous vehicles. In this paper, we propose a data-driven approach for estimating the driver's take-over readiness based purely on observable cues from invehicle vision sensors. We present an extensive naturalistic drive dataset of drivers in a conditionally autonomous vehicle running on Californian freeways. We collect subjective ratings for the driver's take-over readiness from multiple human observers viewing the sensor feed. Analysis of the ratings in terms of intra-class correlation coefficients (ICCs) shows a high degree of consistency in the ratings across raters. We define a metric for the driver's take-over readiness termed the 'Observable Readiness Index (ORI)' based on the ratings. Finally, we propose an LSTM model for continuous estimation of the driver's ORI based on a holistic representation of the driver's state, capturing gaze, hand, pose and foot activity. Our model estimates the ORI with a mean absolute error of 0.449 on a 5 point scale.

Tradeoffs between safe/comfortable headways versus mobility-enhancing headways in an automated driving environment: preliminary insights using a driving simulator experiment

“…These results could be explained by the previous findings that the intention to take over the automated driving mode is associated with the trust in AI technology (Deo and Trivedi, 2019;Hengstler et al, 2016;Miller et al, 2016;Petersen et al, 2019). Molnar et al (2018) pointed out that the trust in automated driving and the acceptance of technology would influence the decision of FEBE transition between automated mode and manual mode.…”

Deep CNN, Body Pose, and Body-Object Interaction Features for Drivers’ Activity Monitoring

“…As a result, the model is unsuitable for real-time applications. Likewise, Deo and Trivedi [4] suggest an LSTM-based deep model for continuous estimation of the drivers take-over readiness and is based on a holistic representation of the drivers state, gaze, hand, pose and foot activity. The approach is similar to [10] in the sense that it requires information from various modules (e.g.…”

“…It is a key component of knowing how vehicles will learn to adapt to various driving conditions and environments. To address this, recent research on recognising basic driver's actions such as eating, drinking, interacting with the vehicle controls, and so on [3], [4], [5], [6], [7], [8], is only the first step. This study advances this by proposing a novel approach to enhance the performance of the automatic recognition of driver's activities from still images captured by vehicle cameras.…”

“…Driver action recognition is often focused on vital cues such as head/body pose of the driver, and their interaction with objects in a given scene. Most of the recent approaches [6], [10], [3], [4], [11], [5] focus selectively on these vital cues to improve the recognition accuracy. In machine learning, this kind of processes is often referred to as the attention mechanism.…”

Forced Spatial Attention for Driver Foot Activity Classification

“…As can be inferred from the individual class identities, the desired output changes based on where the driver's foot is in the image. We chose these classes as they are good indicators of a driver's preparatory motion, and are also strongly tied to the time it takes for a driver completely regain control of the car from an autonomous agent [8,9] -also known as the takeover time. Figure 1 depicts the goal of this study and its role in solving the bigger problem of driver vigilance and takeover time estimation.…”