Real‐Time Non‐Driving Behavior Recognition Using Deep Learning‐Assisted Triboelectric Sensors in Conditionally Automated Driving

Abstract

Real-time recognition of non-driving behaviors is of great importance in conditionally automated driving, as it determines the takeover time budget, which in turn has a huge impact on the performance of the takeover. Here, a novel real-time non-driving behavior recognition system (RNBRS) integrating self-powered, low-cost, easy-to-manufacture triboelectric sensors, and a deep learning model is proposed. The structure, working mechanism, and electrical characteristics of triboelectric sensors are investigated and analyzed. Through the ingenious structural design of single-electrode triboelectric sensors and driving simulation experiments under conditional automated driving, nondriving behaviors are captured in the form of electrical signals. A well-trained long short-term memory network model is adopted to recognize the five most typical non-driving behaviors, including phone, console touchpad, driving, monitoring driving, and no operation, and test accuracy of 93.5% is achieved. Demonstration of a set of controlled experiments shows that RNBRS enables vehicles with conditional automation to dynamically adjust takeover time budget based on driver behavior, therefore significantly improving both safety and stability of takeover. This study opens new frontiers for the development of self-powered electronics and inspires new thoughts on human-machine interaction and the safety of autonomous vehicles.

Triboelectric Nanogenerator Assisted by Machine Learning

“…Accurately and sensitively identifying the driver's nondriving behaviors, such as talking, touching the console trackpad, etc., is of great significance for realizing conditional automatic driving, avoiding traffic accidents, and ensuring personal safety. Zhang et al 21 designed a real-time nondriving behavior recognition system (RNBRS) based on triboelectric sensors (Figure 9a). The system has the characteristics of integrated self-powered triboelectric sensor, low cost, and easy manufacturing.…”

“…(c) Driver driving behavior information is input into the deep learning model. (d) Accuracy of RNBRS in identifying driving behavior …”

Synergizing Machine Learning Algorithm with Triboelectric Nanogenerators for Advanced Self-Powered Sensing Systems

“…The LSTM model has also been applied in capturing non-driving behaviors. Zhang et al proposed a real-time non-driving behavior recognition system [ 170 ]. As shown in Figure 9 c(i), the triboelectric sensor data and image data of five types of non-driving behaviors of eight drivers were collected and analyzed using a clever single-electrode structure.…”

Deep-Learning-Assisted Sensor with Multiple Perception Capabilities for an Intelligent Driver Assistance Monitoring System

“…With the rapid development of the Internet of Things (IoT) and artificial intelligence, 1,2 the global interest in autonomous driving technology has significantly increased. 3 However, due to the limitations of the current technology and the debate over traffic safety ethics, fully autonomous driving has not been achieved yet and could not be promoted in the short term. In view of that, a driver assistance system has been developed, and it represents a future trend of vehicle development.…”