RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) aims to find near-optimal policies from logged data without further environment interaction. Model-based algorithms, which learn a model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem. In this work, we present Robust Adversarial Model-Based Offline RL (RAMBO), a novel approach to model-based offline RL. To achieve conservatism, we formulate the problem as a two-player zero sum game against an adversarial environment model. The model is trained minimise the value function while still accurately predicting the transitions in the dataset, forcing the policy to act conservatively in areas not covered by the dataset. To approximately solve the two-player game, we alternate between optimising the policy and optimising the model adversarially. The problem formulation that we address is theoretically grounded, resulting in a PAC performance guarantee and a pessimistic value function which lower bounds the value function in the true environment. We evaluate our approach on widely studied offline RL benchmarks, and demonstrate that our approach achieves state of the art performance.

Destabilizing Attack and Robust Defense for Inverter-Based Microgrids by Adversarial Deep Reinforcement Learning

“…Perturbing the training environment was previously discussed in unsupervised environment design [6,17], domain randomization [27,36], robust adversarial RL [28,33] and risk aversion [10,25]. However, their focus on robustness differs from our perspective.…”

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

“…The second category consists of model-based methods such as adversarial model learning (Rigter et al, 2022), learning pessimistic models (Kidambi et al, 2020;Guo et al, 2022), using model ensembles to form penalties (Yu et al, 2020), or combining model and values .…”

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

“…We additionally compared the proposed approach with RAMBO [61], a concurrent work that also formulates offline RL as a two-player zero-sum game. The results of RAMBO for random, medium, medium-expert and medium-replay are taken from [61].…”

“…We additionally compared the proposed approach with RAMBO [61], a concurrent work that also formulates offline RL as a two-player zero-sum game. The results of RAMBO for random, medium, medium-expert and medium-replay are taken from [61]. For the other two dataset types, we run the official code and follow the hyperparameter search procedure reported in its paper.…”