MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction With Automation

Abstract

Today, and possibly for a long time to come, the full driving task is too complex an activity to be fully formalized as a sensing-acting robotics system that can be explicitly solved through model-based and learning-based approaches in order to achieve full unconstrained vehicle autonomy. Localization, mapping, scene perception, vehicle control, trajectory optimization, and higher-level planning decisions associated with autonomous vehicle development remain full of open challenges. This is especially true for unconstrained, real-world operation where the margin of allowable error is extremely small and the number of edge-cases is extremely large. Until these problems are solved, human beings will remain an integral part of the driving task, monitoring the AI system as it performs anywhere from just over 0% to just under 100% of the driving. The governing objectives of the MIT Advanced Vehicle Technology (MIT-AVT) study are to (1) undertake large-scale real-world driving data collection that includes high-definition video to fuel the development of deep learning based internal and external perception systems, (2) gain a holistic understanding of how human beings interact with vehicle automation technology by integrating video data with vehicle state data, driver characteristics, mental models, and self-reported experiences with technology, and (3) identify how technology and other factors related to automation adoption and use can be improved in ways that save lives. In pursuing these objectives, we have instrumented 23 Tesla Model S and Model X vehicles, 2 Volvo S90 vehicles, 2 Range Rover Evoque, and 2 Cadillac CT6 vehicles for both long-term (over a year per driver) and medium term (one month per driver) naturalistic driving data collection. Furthermore, we are continually developing new methods for analysis of the massive-scale dataset collected from the instrumented vehicle fleet. The recorded data streams include IMU, GPS, CAN messages, and high-definition video streams of the driver face, the driver cabin, the forward roadway, and the instrument cluster (on select vehicles). The study is on-going and growing. To date, we have 122 participants, 15,610 days of participation, 511,638 miles, and 7.1 billion video frames. This paper presents the design of the study, the data collection hardware, the processing of the data, and the computer vision algorithms currently being used to extract actionable knowledge from the data. 01 231 4523 67 89 8 %& 'ÿ )*+ ,-,.,*/ÿ 0123 45 1 '142-,5 ,67ÿ 8+ *97 :; <=>ÿ @AB; CDÿ ; AE =F; GH ÿIJ KFL; M NM OFB; ÿ =F>DH ÿPQR SPT ULM VGLDH ÿPWW XGCM NY GDH ÿWZ [M Y GDÿ =LM VGBH ÿQPPR SI\ XM =GAÿ ] LF@GDH ÿJPPÿa b b a cd :; <=>ÿ =F; Fÿ NAY Y GN; M ABÿ M Dÿ ABeAM Bef ÿ :; F; M D; M NDÿ <O=F; G=ÿ ABH ÿ ghiÿ klm ÿ knopf

A survey of deep learning techniques for autonomous driving

“…End2End control papers mainly employ either deep neural networks trained offline on real‐world and/or synthetic data (Bechtel et al, ; Bojarski et al, ; C. Chen, Seff, Kornhauser, & Xiao, ; Eraqi et al, ; Fridman et al, ; Hecker et al, ; Rausch et al, ; Xu et al, ; S. Yang et al, ), or DRL systems trained and evaluated in simulation (Jaritz et al, ; Perot, Jaritz, Toromanoff, & Charette, ; Sallab et al, 2017b). Methods for porting simulation trained DRL models to real‐world driving have also been reported (Wayve, 2018), as well as DRL systems trained directly on real‐world image data (Pan et al, , ).…”

“…In Fridman et al (), the authors present a study that seeks to collect and analyze large‐scale naturalistic data of semi‐autonomous driving to better characterize the state‐of‐the‐art of the current technology. The study involved participants, vehicles, miles, and approximately billion video frames.…”

“…The reason may be F I G U R E 7 Sensor suite of the nuTonomy ® self-driving car (Caesar et al, 2019) [Color figure can be viewed at wileyonlinelibrary.com] GRIGORESCU ET AL. In Fridman et al (2017), the authors present a study that seeks to collect and analyze large-scale naturalistic data of semi-autonomous driving to better characterize the state-of-the-art of the current technology. The study involved 99 participants, 29 vehicles, 405, 807 miles, and approximately 5.5 billion video frames.…”

“…This kind of processing allows the training of large and complex network architectures, which in turn require huge amounts of training samples (see Section 8).End2End control papers mainly employ either deep neural networks trained offline on real-world and/or synthetic data(Bechtel et al, 2018;Bojarski et al, 2016;C. Chen, Seff, Kornhauser, & Xiao, 2015;Eraqi et al, 2017;Fridman et al, 2017;Hecker et al, 2018;Rausch et al, 2017;Xu et al, 2017;S. Yang et al, 2017a), or DRL systems trained and evaluated in simulation(Jaritz et al, 2018;Perot, Jaritz, Toromanoff, & Charette, 2017;Sallab et al, 2017b).…”

Mining the Automotive Industry: A Network Analysis of Corporate Positioning and Technological Trends

“…Change in the automotive industry has been intensively investigated in past studies [9,14,15,16,17], focusing on changing geographies and challenges of the technological transformation, and on technical, environmental, and management implications. Moreover, network analysis has been applied in the field, mainly to understand Supply-Chain systems in the automotive industry [18,19].…”