Stochastic first-order methods for average-reward Markov decision processes

Abstract

We study the problem of average-reward Markov decision processes (AMDPs) and develop novel first-order methods with strong theoretical guarantees for both policy evaluation and optimization. Existing on-policy evaluation methods suffer from sub-optimal convergence rates as well as failure in handling insufficiently random policies, e.g., deterministic policies, for lack of exploration. To remedy these issues, we develop a novel variance-reduced temporal difference (VRTD) method with linear function approximation for randomized policies along with optimal convergence guarantees, and an exploratory variance-reduced temporal difference (EVRTD) method for insufficiently random policies with comparable convergence guarantees. We further establish linear convergence rate on the bias of policy evaluation, which is essential for improving the overall sample complexity of policy optimization. On the other hand, compared with intensive research interest in finite sample analysis of policy gradient methods for discounted MDPs, existing studies on policy gradient methods for AMDPs mostly focus on regret bounds under restrictive assumptions on the underlying Markov processes (see, e.g., Abbasi-Yadkori et al., 2019), and they often lack guarantees on the overall sample complexities. Towards this end, we develop an average-reward variant of the stochastic policy mirror descent (SPMD) (Lan, 2022). We establish the first O( −2 ) sample complexity for solving AMDPs with policy gradient method under both the generative model (with unichain assumption) and Markovian noise model (with ergodic assumption). This bound can be further improved to O( −1 ) for solving regularized AMDPs. Our theoretical advantages are corroborated by numerical experiments.

Policy Mirror Descent Inherently Explores Action Space

“…As we have discussed in Section 1, there have been two main approaches in the literature of online PG methods for addressing (2.4). One approach forces (2.3) with myopic exploration, with either ǫ-exploration [4,17], or policy perturbation [26]. A drawback of myopic exploration, aside from another layer of complexity in implementation, is its repeating sample of non-optimal or high-risk actions, even if the policy has identified these actions.…”

“…It is worth noting that the two-step construction of VBE-I also makes it feasible to utilize other online policy evaluation operator for constructing V π,ξV with linearly converging bias. This is particularly helpful as it can allow the policy evaluation step to incorporate simple function approximation for problems with large state spaces (e.g., linear approximation, [19,26]). This also seems to be a unique feature of VBE-I operator compared to other policy evaluation operators studied in this manuscript.…”

“…O(1/ǫ 6 )) is obtained. A more delicate method, based on policy perturbation, that periodically switches the policy to an exploring one within policy evaluation [26], attains an O(1/ǫ 2 ) sample complexity. We refer to this type of explicit exploration as myopic exploration, since it ignores the cause for a particular action to have small probability.…”

Data-Driven Robust Multi-Agent Reinforcement Learning

“…In this section, we study the problem of optimal control, which aims to find a policy that optimizes the robust averagereward: π * = arg max π g π P . Similar to Assumption 1, we make the following assumption to guarantee that the average-reward is independent of the initial state (Abounadi et al, 2001;Li et al, 2022). Assumption 4.…”