Planning for Autonomous Cars that Leverage Effects on Human Actions

Abstract

Traditionally, autonomous cars make predictions about other drivers' future trajectories, and plan to stay out of their way. This tends to result in defensive and opaque behaviors. Our key insight is that an autonomous car's actions will actually affect what other cars will do in response, whether the car is aware of it or not. Our thesis is that we can leverage these responses to plan more efficient and communicative behaviors. We model the interaction between an autonomous car and a human driver as a dynamical system, in which the robot's actions have immediate consequences on the state of the car, but also on human actions. We model these consequences by approximating the human as an optimal planner, with a reward function that we acquire through Inverse Reinforcement Learning. When the robot plans with this reward function in this dynamical system, it comes up with actions that purposefully change human state: it merges in front of a human to get them to slow down or to reach its own goal faster; it blocks two lanes to get them to switch to a third lane; or it backs up slightly at an intersection to get them to proceed first. Such behaviors arise from the optimization, without relying on hand-coded signaling strategies and without ever explicitly modeling communication. Our user study results suggest that the robot is indeed capable of eliciting desired changes in human state by planning using this dynamical system.

Planning and Decision-Making for Autonomous Vehicles

“…It is a common pattern to model other vehicles' behavior as expected utility maximizing-i.e., an agent is expected to execute the most beneficial controls (87). Therefore, a reward or utility function needs to be known or learned.…”

Active Preference-Based Learning of Reward Functions

“…Approach Overview. Inverse Reinforcement Learning (IRL) [15,19,23] enables us to learn R H through demonstrated trajectories. However, IRL requires the human to show demonstrations of the optimal sequence of actions.…”

Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models

“…Future experiments will focus on more realistic driving scenarios and learning risk preferences of human UAV pilots in cluttered environments. Finally, we plan on studying the game theoretic IRL setting (e.g., [43,48]), where multiple risk-sensitive agents interact.…”

“…In order to realize this vision, robots must be able to (1) accurately predict the actions of humans in their environment, (2) quickly learn the preferences of human agents in their proximity and act accordingly, and (3) learn how to accomplish new tasks from human demonstrations. Inverse Reinforcement Learning (IRL) [41,32,2,29,38,50,16] is a powerful and flexible framework for tackling these challenges and has been previously used for tasks such as modeling and mimicking human driver behavior [1,28,43], pedestrian trajectory prediction [51,31], and legged robot locomotion [52,27,35]. The underlying assumption behind IRL is that humans act optimally with respect to an (unknown) cost function.…”