Apprenticeship learning via inverse reinforcement learning

Abstract

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function.

Learning from Demonstrations: Is It Worth Estimating a Reward Function?

“…These two methods are Apprenticeship Learning (AL) [1] and Inverse Reinforcement Learning (IRL) [8]. In the AL framework, the agent tries to learn the expert policy or at least a policy which is as good as the expert policy (according to an unknown reward function).…”

“…AL can be reduced to classification [7,3,6,11] where the agent tries to mimic the expert policy via a Supervised Learning (SL) method such as classification. There exist also several AL algorithms inspired by IRL such as [1,10] but they need to solve recursively MDPs which is a difficult problem when the state space is large and the dynamics of the MDP is unknown.…”

Generalizing Hyper-heuristics via Apprenticeship Learning

“…In this paper, we describe a method to take policy matrices learned on small instances and generalise them to apply to different instances, and with the particular aim to apply them to larger instances. We used a form of apprenticeship learning (a.k.a learning by demonstration or imitation learning) [1] for generalizing the demonstrations provided by an expert. Apprenticeship learning has a wide range of applications in control and robotics and is heavily based on Inverse Reinforcement Learning (IRL).…”

Apprenticeship Learning of Ship Behavior in Crowded Area by Dimension Compression

“…Shirai et al analyzed differences in ship behavior in Tokyo Bay by categorizing the sizes of ships and constructing a traffic flow network [6]. However, the model they created is not useful for real-time forecasting.…”