Off-Policy Meta-Reinforcement Learning With Belief-Based Task Inference

Abstract

Meta-reinforcement learning (RL) addresses the problem of sample inefficiency in deep RL by using experience obtained in past tasks for solving a new task. However, most existing meta-RL methods require partially or fully on-policy data, which hinders the improvement of sample efficiency. To alleviate this problem, we propose a novel off-policy meta-RL method, embedding learning and uncertainty evaluation (ELUE). An ELUE agent is characterized by the learning of what we call a task embedding space, an embedding space for representing the features of tasks. The agent learns a belief model over the task embedding space and trains a belief-conditional policy and Q-function. The belief model is designed to be agnostic to the order in which task information is obtained, thereby reducing the difficulty of task embedding learning. For a new task, the ELUE agent collects data by the pretrained policy, and updates its belief on the basis of the belief model. Thanks to the belief update, the performance of the agent improves with a small amount of data. In addition, the agent updates the parameters of its policy and Q-function so that it can adjust the pretrained relationships when there are enough data. We demonstrate that ELUE outperforms state-of-the-art meta RL methods through experiments on meta-RL benchmarks.

A Survey of Meta-Reinforcement Learning

“…Some methods use both PPG and black-box components [252,187]. In particular, even when training a fully black-box method, the policy or inner-loop can be fine-tuned with policy gradients at meta-test time [116,252,97].…”

“…Finally, we conclude with a discussion of the trade-offs concerning task inference methods. Table 2 summarizes these categories and task inference methods [95,185,104,213,269,281,64,130,282,17,97].…”

“…It can be particularly useful to represent the task not just as a sample from this latent variable, but also as the full distribution of the latent variable, e.g., its mean and variance, in order to capture uncertainty about the task [281]. Additionally, Rakelly et al [185] make use of the Markov property to represent the task distribution conditioned on D as a product of individual distributions conditioned on each transition in D. More generally, this entails the exchangeability, or permutation invariance, of the transitions, which has been exploited by other meta-RL methods as well [67,16,110,97,162,241], using representations such as neural processes [71] and transformers [226]. Moreover, permutation invariance at the level of episodes has also been used [116].…”

“…On the other hand, we want to utilize as much useful information from the learned inductive bias as possible to improve learning efficiency on OOD tasks. Although recent works investigate how to improve generalization [116,129,77,252,86,97,1] or adaptation efficiency [51,142,120] on OOD tasks with small distribution shifts, more work remains to be done on how to adaptively handle larger distribution shifts between training and test tasks.…”

“…However, these approaches both allow for additional online adaptation after the offline adaptation in the inner-loop, and the offline data either contains only observations [38], or conditions on additional expert actions [178]. One method uses permutation-invariant memory to enable an off-policy inner-loop, but only evaluates with data collected from prior policies [97]. Consequently, offline RL via online RL remains an interesting setting for future work with the potential for designing more effective offline RL algorithms by meta-learning in both the few-shot and many-shot settings.…”