Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Abstract

Most existing theoretical analysis of reinforcement learning (RL) is limited to the tabular setting or linear models due to the difficulty in dealing with function approximation in high dimensional space with an uncertain environment. This work offers a fresh perspective into this challenge by analyzing RL in a general reproducing kernel Hilbert space (RKHS). We consider a family of Markov decision processes M of which the reward functions lie in the unit ball of an RKHS and transition probabilities lie in a given arbitrary set. We define a quantity called perturbational complexity by distribution mismatch ∆ M (ǫ) to characterize the complexity of the admissible state-action distribution space in response to a perturbation in the RKHS with scale ǫ. We show that ∆ M (ǫ) gives both the lower bound of the error of all possible algorithms and the upper bound of two specific algorithms (fitted reward and fitted Q-iteration) for the RL problem. Hence, the decay of ∆ M (ǫ) with respect to ǫ measures the difficulty of the RL problem on M. We further provide some concrete examples and discuss whether ∆ M (ǫ) decays fast or not in these examples. As a byproduct, we show that when the reward functions lie in a high dimensional RKHS, even if the transition probability is known and the action space is finite, it is still possible for RL problems to suffer from the curse of dimensionality.

Reinforcement Learning with Function Approximation: From Linear to Nonlinear

“…This quantity can give both the lower bound and upper bound of the sample complexity of these RL problems and hence measure their difficulty. Moreover, both fast eigenvalue decay and finite concentration coefficient can lead to small perturbational complexity by distribution mismatch [35, Proposition 2 and 3] and hence results in [35] generalize both categories of the previous results in the nonlinear setting.…”

“…To better capture the influence of the distribution mismatch in the RL problem, [35] introduce a quantity called perturbational complexity by distribution mismatch for a large class of the RL problems in the nonlinear setting when a generative model is accessible. This quantity can give both the lower bound and upper bound of the sample complexity of these RL problems and hence measure their difficulty.…”

“…(ii) Given a Banach space, a positive constant ǫ > 0 and a probability distribution ν ∈ P(S × A), we define B ǫ,ν , a ν-perturbation space with scale ǫ, as follows: In the generative model setting, [35] prove that any RL algorithm J n : θ → R on M with at most n accesses to the generative model satisfies that Therefore, the perturbational complexity of distribution mismatch gives a lower bound for RL problems on M. Note that in [35], it is assumed that we can only obtain a noisy reward with a standard normal noise in the generative model, rather than the exact reward. On the other hand, if B is the Barron space or an RKHS, then the output πθ of Algorithm 4 satisfies:…”

“…Relatively simple function approximation methods, such as the linear model in [53,27] or generalized linear model in [52,34] have been examined in the context of RL algorithms. Meanwhile, nonlinear forms like kernel approximation [14,54,55,36,35] are also studied in RL problems to further bridge the gap between theoretical results under restrictive assumptions and practice.…”

“…This challenge reveals one essential difficulty of RL problems with nonlinear function approximation and hence additional assumptions are needed to derive meaningful results for nonlinear RL in the literature, including assumptions on the fast eigenvalue decay of the kernel and assumptions on the finite concentration coefficient. We finally introduce the perturbational complexity by distribution mismatch in [35], which quantifies the difficulty of a large class of the RL problems in the nonlinear setting as it can give both lower bound and upper bound of the sample complexity of these RL problems. Directions for future work are discussed in Section 7.…”