Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models

Majumdar, Anirudha; Singh, Sumeet; Mandlekar, Ajay; Pavone, Marco

doi:10.15607/rss.2017.xiii.069

Cited by 37 publications

(58 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One possibility is to learn a distortion risk metric that explains how humans evaluate risk in the given application domain and then employ the learned risk metric. We describe first steps towards this in [20], where we have introduced a framework for risk-sensitive inverse reinforcement learning for learning humans' risk preferences from the class of coherent risk metrics.…”

Section: Discussionmentioning

confidence: 99%

How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics

Majumdar

Pavone

2019

Springer Proceedings in Advanced Robotics

Self Cite

157

145

View full text Add to dashboard Cite

Endowing robots with the capability of assessing risk and making riskaware decisions is widely considered a key step toward ensuring safety for robots operating under uncertainty. But, how should a robot quantify risk? A natural and common approach is to consider the framework whereby costs are assigned to stochastic outcomes-an assignment captured by a cost random variable. Quantifying risk then corresponds to evaluating a risk metric, i.e., a mapping from the cost random variable to a real number. Yet, the question of what constitutes a "good" risk metric has received little attention within the robotics community. The goal of this paper is to explore and partially address this question by advocating axioms that risk metrics in robotics applications should satisfy in order to be employed as rational assessments of risk. We discuss general representation theorems that precisely characterize the class of metrics that satisfy these axioms (referred to as distortion risk metrics), and provide instantiations that can be used in applications. We further discuss pitfalls of commonly used risk metrics in robotics, and discuss additional properties that one must consider in sequential decision making tasks. Our hope is that the ideas presented here will lead to a foundational framework for quantifying risk (and hence safety) in robotics applications.

show abstract

Section: Discussionmentioning

confidence: 99%

How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics

Majumdar

Pavone

2019

Springer Proceedings in Advanced Robotics

Self Cite

157

145

View full text Add to dashboard Cite

show abstract

“…Since such instrumentation needs to be done for any new task that we may wish to learn, it poses a significant bottleneck to widespread adoption of reinforcement learning for robotics, and precludes the use of these methods directly in open-world environments that lack this instrumentation. Data-driven approaches for reward specification [29,1,11,17,12,48,33,9,27,4,18] seek to overcome this issue, but typically require demonstration data to acquire rewards. Such data can be onerous and time-consuming for users to provide.…”

Section: Related Workmentioning

confidence: 99%

End-To-End Robotic Reinforcement Learning without Reward Engineering

Singh¹,

Yang²,

Hartikainen³

et al. 2019

Robotics: Science and Systems XV

165

View full text Add to dashboard Cite

The combination of deep neural network models and reinforcement learning algorithms can make it possible to learn policies for robotic behaviors that directly read in raw sensory inputs, such as camera images, effectively subsuming both estimation and control into one model. However, realworld applications of reinforcement learning must specify the goal of the task by means of a manually programmed reward function, which in practice requires either designing the very same perception pipeline that end-to-end reinforcement learning promises to avoid, or else instrumenting the environment with additional sensors to determine if the task has been performed successfully. In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task. While requesting labels for every single state would amount to asking the user to manually provide the reward signal, our method requires labels for only a tiny fraction of the states seen during training, making it an efficient and practical approach for learning skills without manually engineered rewards. We evaluate our method on real-world robotic manipulation tasks where the observations consist of images viewed by the robot's camera. In our experiments, our method effectively learns to arrange objects, place books, and drape cloth, directly from images and without any manually specified reward functions, and with only 1-4 hours of interaction with the real world. Videos of learned behavior are available at sites.

show abstract

“…Consequently, the predicted trajectories might be too conservative to reflect potential dangers in some situations. Although recent works such as [25] and [26] have introduced some "irrational" behaviors into planning-based approaches, it is still not sufficient to cover all human driving behaviors. Moreover, to obtain a solvable and interpretable planning problem, the learned reward/cost functions are typically linear combinations of features.…”

Section: A Motivationmentioning

confidence: 99%

Generic Prediction Architecture Considering both Rational and Irrational Driving Behaviors

Sun

Tomizuka

2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

View full text Add to dashboard Cite

Accurately predicting future behaviors of surrounding vehicles is an essential capability for autonomous vehicles in order to plan safe and feasible trajectories. The behaviors of others, however, are full of uncertainties. Both rational and irrational behaviors exist, and the autonomous vehicles need to be aware of this in their prediction module. The prediction module is also expected to generate reasonable results in the presence of unseen and corner scenarios. Two types of prediction models are typically used to solve the prediction problem: learning-based model and planning-based model. Learning-based model utilizes real driving data to model the human behaviors. Depending on the structure of the data, learning-based models can predict both rational and irrational behaviors. But the balance between them cannot be customized, which creates challenges in generalizing the prediction results. Planning-based model, on the other hand, usually assumes human as a rational agent, i.e., it anticipates only rational behavior of human drivers. In this paper, a generic prediction architecture is proposed to address various rationalities in human behavior. We leverage the advantages from both learningbased and planning-based prediction models. The proposed approach is able to predict continuous trajectories that wellreflect possible future situations of other drivers. Moreover, the prediction performance remains stable under various unseen driving scenarios. A case study under a real-world roundabout scenario is provided to demonstrate the performance and capability of the proposed prediction architecture.

show abstract

Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models

Cited by 37 publications

References 44 publications

How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics

How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics

End-To-End Robotic Reinforcement Learning without Reward Engineering

Generic Prediction Architecture Considering both Rational and Irrational Driving Behaviors

Contact Info

Product

Resources

About