Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach

Abstract

We propose a black-box reduction that turns a certain reinforcement learning algorithm with optimal regret in a (near-)stationary environment into another algorithm with optimal dynamic regret in a non-stationary environment, importantly without any prior knowledge on the degree of non-stationarity. By plugging different algorithms into our black-box, we provide a list of examples showing that our approach not only recovers recent results for (contextual) multi-armed bandits achieved by very specialized algorithms, but also significantly improves the state of the art for linear bandits, episodic MDPs, and infinite-horizon MDPs in various ways. Specifically, in most cases our algorithm achieves the optimal dynamic regret O(min{ √ LT , ∆ 1/3 T 2/3 }) where T is the number of rounds and L and ∆ are the number and amount of changes of the world respectively, while previous works only obtain suboptimal bounds and/or require the knowledge of L and ∆.

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

“…Non-stationary RL. Non-stationary RL has been mostly studied in the unconstrained setting (Jaksch et al, 2010;Auer et al, 2019;Ortner et al, 2020;Domingues et al, 2021;Mao et al, 2020;Zhou et al, 2020;Touati & Vincent, 2020;Fei et al, 2020;Zhong et al, 2021;Cheung et al, 2020;Wei & Luo, 2021). Our work is related to policy-based methods for non-stationary RL since the optimal solution of CMDP is usually a stochastic policy (Altman, 1999) and thus a policy-based method is preferred.…”

“…To eliminate the assumption of having prior knowledge on variation budgets, Wei & Luo (2021) recently outline that an adaptive restart approach can be used to convert any upper-confidence-bound-type stationary RL algorithm to a dynamic-regret-minimizing algorithm. However, this approach is proposed only for the unconstrained problems and relies on the assumption of having an optimistic estimator of the optimal value function.…”

“…Thus, their approach may not be directly applicable to our CMDP setting where a policy-based method is preferred. We leave it as an important open problem to handle the case without the prior knowledge on variation budgets and generalize the nonstationarity detection mechanism in Wei & Luo (2021) to our CMDP setting.…”

“…This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration. An interesting future direction is to relax the assumption on the prior knowledge of the variation budgets and generalize the non-stationarity detection mechanism inWei & Luo (2021) to our CMDP setting. Zhong, H., Yang, Z., and Szepesvári, Z. W. C. Optimistic policy optimization is provably efficient in nonstationary MDPs.…”

Dynamic Regret Minimization for Control of Non-stationary Linear Dynamical Systems

“…Second, the probabilities with which the exploration is carried out are different for the LQR problem owing to the quadratic cost. More recently, the authors in Wei and Luo [2021] outline that for many classes of episodic reinforcement learning problems, a similar strategy can be used to convert any Upper Confidence Bound (UCB) type stationary reinforcement learning algorithm to a dynamic regret minimizing algorithm. There are quite a few differences between Wei and Luo [2021] and our work: the LQR problem is not covered by the classes of MDPs they consider, we look at a non-episodic version of the LQR problem, and our algorithm is certainty equivalent controller-based and not a UCB-type.…”