Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning

Abstract

In the field of reinforcement learning there has been recent progress towards safety and high-confidence bounds on policy performance. However, to our knowledge, no practical methods exist for determining high-confidence policy performance bounds in the inverse reinforcement learning setting---where the true reward function is unknown and only samples of expert behavior are given. We propose a sampling method based on Bayesian inverse reinforcement learning that uses demonstrations to determine practical high-confidence upper bounds on the alpha-worst-case difference in expected return between any evaluation policy and the optimal policy under the expert's unknown reward function. We evaluate our proposed bound on both a standard grid navigation task and a simulated driving task and achieve tighter and more accurate bounds than a feature count-based baseline. We also give examples of how our proposed bound can be utilized to perform risk-aware policy selection and risk-aware policy improvement. Because our proposed bound requires several orders of magnitude fewer demonstrations than existing high-confidence bounds, it is the first practical method that allows agents that learn from demonstration to express confidence in the quality of their learned policy.

Recent Advances in Robot Learning from Demonstration

“…Since LfD utilizes expert demonstrations, the robot can be better incentivized to stay within safe or relevant regions of the state space, especially when compared with techniques that require significant exploration, such as reinforcement learning. This is because demonstrations provide a way to assess the safety or risk associated with regions of the state space (e.g., [196][197][198]. Furthermore, several LfD methods provide and utilize measures of uncertainty associated with different parts of the state space (e.g., 62, 81, 100), enabling communication of the system's confidence to the user.…”

Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review

“…In contrast, the ML-IRL is based on maximum likelihood estimation, which cannot incorporate prior knowledge and handle uncertainty. IRL with the Bayesian optimization method has been used to learn driving strategies [54], mobile robot navigation [55,56], and robot demonstrative learning [57] with good performance. Hierarchical BIRL extended on the original basis outperforms MaxEnt-IRL in cab driver route selection based on maps and GPS data [58].…”

Hierarchical Interpretable Imitation Learning for End-to-End Autonomous Driving

“…Furthermore, Brown et al [54] construct a sampling-based Bayesian IRL model, which utilizes expert trajectories to calculate practical high-confidence upper bounds on the αworst-case difference in expected return under the unseen scenarios without a reward function. Palan et al [55] propose DemPref model, which utilizes the expert trajectory to learn a coarse reward function, the trajectory is used to ground the (active) query generation process, to improve the quality of the generated queries.…”