Perturbation theory and uniform ergodicity for discrete-time Markov chains

Abstract

We study perturbation theory and uniform ergodicity for discrete-time Markov chains on general state spaces in terms of the uniform moments of the first hitting times on some set. The methods we adopt are different from previous ones. For reversible and non-negative definite Markov chains, we first investigate the geometrically ergodic convergence rates. Based on the estimates, together with a first passage formula, we then get the convergence rates in uniform ergodicity. If the transition kernel P is only reversible, we transfer to study the two-skeleton chain with the transition kernel P 2 . At a technical level, the crucial point is to connect the geometric moments of the first return times between P and P 2 .

Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings

“…Lastly, our analysis relies heavily on the theory of ergodicity for MDPs. We build on the works of Mitrophanov (2005) which yields perturbation bounds on the state distribution, and subsequent improvements in the assumptions and condition numbers (Ferré, Hervé, and Ledoux 2013;Rudolf, Schweizer et al 2018;Mao and Song 2020).…”