Modeling risk anticipation and defensive driving on residential roads with inverse reinforcement learning

Shimosaka, Masamichi; Kaneko, Takuhiro; Nishi, Koichi

doi:10.1109/itsc.2014.6957937

Cited by 27 publications

(16 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our implementation, we learn this reward function from human data. We collect demonstrations of a driver in a simulation environment, and use Inverse Reinforcement Learning (Ng et al 2000;Abbeel and Ng 2005;Ziebart et al 2008;Levine and Koltun 2012;Shimosaka et al 2014;Kuderer et al 2015) to recover a reward function that explains the demonstrations.…”

Section: Offline Estimation Of Human Reward Parametersmentioning

confidence: 99%

Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state

et al. 2018

View full text Add to dashboard Cite

Traditionally, autonomous cars treat human-driven vehicles like moving obstacles. They predict their future trajectories and plan to stay out of their way. While physically safe, this results in defensive and opaque behaviors. In reality, an autonomous car's actions will actually affect what other cars will do in response, creating an opportunity for coordination. Our thesis is that we can leverage these responses to plan more efficient and communicative behaviors. We introduce a formulation of interaction with human-driven vehicles as an underactuated dynamical system, in which the robot's actions have consequences on the state of the autonomous car, but also on the human actions and thus the state of the human-driven car. We model these consequences by approximating the human's actions as (noisily) optimal with respect to some utility function. The robot uses the human actions as observations of her underlying utility function parameters. We first explore learning these parameters offline, and show that a robot planning in the resulting underactuated system is more efficient than when treating the person as a moving obstacle. We also show that the robot can target specific desired effects, like getting the person to switch lanes or to proceed first through an intersection. We then explore estimating these parameters online, and enable the robot to perform active information gathering: generating actions that purposefully probe the human in order to clarify their underlying utility parameters, like driving style or attention level. We show that this significantly outperforms passive estimation and improves efficiency. Planning in our model results in coordination behaviors: the robot inches forward at an intersection to see if can go through, or it reverses to make the other car proceed first. These behaviors result from the optimization, without relying on hand-coded signaling strategies. Our user studies support the utility of our model when interacting with real users. Keywords Planning for human-robot interaction • Mathematical models of human behavior • Autonomous driving This is one of several papers published in Autonomous Robots comprising the "Special Issue on Robotics Science and Systems".

show abstract

Section: Offline Estimation Of Human Reward Parametersmentioning

confidence: 99%

Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state

et al. 2018

View full text Add to dashboard Cite

show abstract

“…In our implementation, we learn this reward function from human data. We collect demonstrations of a driver in a simulation environment, and use Inverse Reinforcement Learning [1,12,15,19,23,29] to recover a reward function that explains the demonstrations.…”

Section: E Human Driver Rewardmentioning

confidence: 99%

Planning for Autonomous Cars that Leverage Effects on Human Actions

Sadigh¹,

Sastry²,

Seshia³

et al.

Robotics: Science and Systems XII

371

338

View full text Add to dashboard Cite

Traditionally, autonomous cars make predictions about other drivers' future trajectories, and plan to stay out of their way. This tends to result in defensive and opaque behaviors. Our key insight is that an autonomous car's actions will actually affect what other cars will do in response, whether the car is aware of it or not. Our thesis is that we can leverage these responses to plan more efficient and communicative behaviors. We model the interaction between an autonomous car and a human driver as a dynamical system, in which the robot's actions have immediate consequences on the state of the car, but also on human actions. We model these consequences by approximating the human as an optimal planner, with a reward function that we acquire through Inverse Reinforcement Learning. When the robot plans with this reward function in this dynamical system, it comes up with actions that purposefully change human state: it merges in front of a human to get them to slow down or to reach its own goal faster; it blocks two lanes to get them to switch to a third lane; or it backs up slightly at an intersection to get them to proceed first. Such behaviors arise from the optimization, without relying on hand-coded signaling strategies and without ever explicitly modeling communication. Our user study results suggest that the robot is indeed capable of eliciting desired changes in human state by planning using this dynamical system.

show abstract

“…, b N b , as defined in equation (11). e median of the MHD (MHD 50 ) had been used to evaluate the similarity of simulated and actual trajectories in modeling defensive driving strategies [44] and urban route planning [45].…”

Section: Evaluation Metricsmentioning

confidence: 99%

Learning the Car-following Behavior of Drivers Using Maximum Entropy Deep Inverse Reinforcement Learning

Zhou

Wang

2020

Journal of Advanced Transportation

View full text Add to dashboard Cite

The present study proposes a framework for learning the car-following behavior of drivers based on maximum entropy deep inverse reinforcement learning. The proposed framework enables learning the reward function, which is represented by a fully connected neural network, from driving data, including the speed of the driver’s vehicle, the distance to the leading vehicle, and the relative speed. Data from two field tests with 42 drivers are used. After clustering the participants into aggressive and conservative groups, the car-following data were used to train the proposed model, a fully connected neural network model, and a recurrent neural network model. Adopting the fivefold cross-validation method, the proposed model was proved to have the lowest root mean squared percentage error and modified Hausdorff distance among the different models, exhibiting superior ability for reproducing drivers’ car-following behaviors. Moreover, the proposed model captured the characteristics of different driving styles during car-following scenarios. The learned rewards and strategies were consistent with the demonstrations of the two groups. Inverse reinforcement learning can serve as a new tool to explain and model driving behavior, providing references for the development of human-like autonomous driving models.

show abstract

Modeling risk anticipation and defensive driving on residential roads with inverse reinforcement learning

Cited by 27 publications

References 11 publications

Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state

Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state

Planning for Autonomous Cars that Leverage Effects on Human Actions

Learning the Car-following Behavior of Drivers Using Maximum Entropy Deep Inverse Reinforcement Learning

Contact Info

Product

Resources

About