A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

Abstract

With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

Prioritized offline Goal-swapping Experience Replay

“…In offline reinforcement learning (offline RL), an agent must work with a set of static, pre-existing data rather than interacting with an environment to collect data. This data is often collected by unknown policies Prudencio et al (2022). In the offline goal-conditioned setting, the objective is the same as in online goal-conditioned RL, as defined by Equation 1.…”

“…Training GCRL agents can be difficult due to the sparsity of rewards in GCRL tasks, forcing the agent to explore the environment, which can be unfeasible or even dangerous in some real-world tasks. To utilize RL without environment interactions offline RL allows learning a policy from a dataset without putting real environments at risk ; Prudencio et al (2022). Offline goal-conditioned RL (offline GCRL) combines the generalizability of GCRL and the data-efficiency of offline RL, making it a promising approach for real-world applications Ma et al (2022).…”

“…The first is that the offline dataset only covers a limited state-goal-action space, which can cause incorrect value function estimations for out-of-distribution observations. This can lead to compounding errors in policy deviation ; Prudencio et al (2022). Additionally, each state can have multiple goals, making it hard to learn from a limited state-goal observation space Chebotar et al (2021).…”

An efficient model‐free adaptive optimal control of continuous‐time nonlinear non‐zero‐sum games based on integral reinforcement learning with exploration

“…It is evident that the operation of the integral eliminates the drift dynamics f (x), the utilization of u * i and l i j (x) eliminates the input dynamics g(x). Therefore, based on approximation methods such as neural networks, by solving Equation (10) for each player, the value function V i , Nash equilibrium policy u * i and unknown function l i j can be obtained without requiring any system dynamics. This is the model-free approach in [19] for obtaining the Nash equilibrium solution of CT nonlinear systems.…”

“…Reinforcement learning (RL) has emerged as a popular approach for tackling NZS games. RL is grounded in the principle of trial and error [8], enabling agents to acquire optimal behavioural policies by leveraging feedback responses derived from the environment [9,10]. RL can be broadly categorized into two types: model-free and model-based, with the key distinction being the requirement of dynamic model information [11].…”