Safe Policy Improvement with Baseline Bootstrapping in Factored Environments

Abstract

We present a novel safe reinforcement learning algorithm that exploits the factored dynamics of the environment to become less conservative. We focus on problem settings in which a policy is already running and the interaction with the environment is limited. In order to safely deploy an updated policy, it is necessary to provide a confidence level regarding its expected performance. However, algorithms for safe policy improvement might require a large number of past experiences to become confident enough to change the agent’s behavior. Factored reinforcement learning, on the other hand, is known to make good use of the data provided. It can achieve a better sample complexity by exploiting independence between features of the environment, but it lacks a confidence level. We study how to improve the sample efficiency of the safe policy improvement with baseline bootstrapping algorithm by exploiting the factored structure of the environment. Our main result is a theoretical bound that is linear in the number of parameters of the factored representation instead of the number of states. The empirical analysis shows that our method can improve the policy using a number of samples potentially one order of magnitude smaller than the flat algorithm.

An Interpretable Graph-based Mapping of Trustworthy Machine Learning Research

“…Interestingly, a very recent paper that is not in our analyzed corpus studied the problem of applying fairness constraints on node ranks in a graph [12]. Paper 19 is on safe policy improvement in RL [27] RL refer to policies that maximize expected return in problems where ensuring certain safety constraints is important alongside satisfactory performance [8]. In the context of ML fairness, safe policies can potentially be policies that satisfy equitable performance guarantees for sensitive demographic subgroups.…”

Non-Markovian policies occupancy measures

“…In this section, we study a particular algorithm family called SPIBB for Safe Policy Improvement with Baseline Bootstrapping [23,28,31,40,41,34]. It consists in allowing policy change only when the change is sufficiently supported by the dataset.…”

“…From there, they re-derive AWR. Interestingly, this approximation relies on a construction, given explicitly in their equation (41), which corresponds precisely to our definition of π in their specific setting.…”

Abstraction-Guided Policy Recovery from Expert Demonstrations

“…In order to maximize the performance of this simple model, we instead consider the nearest abstract Cartesian state according to the abstraction function that gave the best performance in experiments (in this case: {x, y}). The Factored Batch RL approach is a method that has been used as a baseline in other offline RL work (Simão and Spaan 2019). It uses complete knowledge of the transition dynamics in the form of a dynamic Bayesian network and calculates the parameters of the model with the offline data.…”