Towards Tight Bounds on the Sample Complexity of Average-reward MDPs

Abstract

We prove new upper and lower bounds for sample complexity of finding an -optimal policy of an infinite-horizon average-reward Markov decision process (MDP) given access to a generative model. When the mixing time of the probability transition matrix of all policies is at most t mix , we provide an algorithm that solves the problem using O(t mix −3 ) (oblivious) samples per state-action pair. Further, we provide a lower bound showing that a linear dependence on t mix is necessary in the worst case for any algorithm which computes oblivious samples. We obtain our results by establishing connections between infinite-horizon average-reward MDPs and discounted MDPs of possible further utility.

Logarithmic regret bounds for continuous-time average-reward Markov decision processes

“…Another popular performance measure is the sample complexity, which is the amount of data required to learn a near-optimal policy; see e.g Brunskill and Li (2014),Jin and Sidford (2021),Wang (2017)…”