An $L^2$ Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximation

Abstract

Reinforcement learning (RL) algorithms based on high-dimensional function approximation have achieved tremendous empirical success in large-scale problems with an enormous number of states. However, most analysis of such algorithms gives rise to error bounds that involve either the number of states or the number of features. This paper considers the situation where the function approximation is made either using the kernel method or the two-layer neural network model, in the context of a fitted Q-iteration algorithm with explicit regularization. We establish an Õ(H 3 |A| 1 4 n − 1 4 ) bound for the optimal policy with Hn samples, where H is the length of each episode and |A| is the size of action space. Our analysis hinges on analyzing the L 2 error of the approximated Q-function using n data points. Even though this result still requires a finite-sized action space, the error bound is independent of the dimensionality of the state space.that we can obtain an Õ((1 − γ) −2 (n − 1 2(1+α) + γ K )) bound for the optimal policy with Kn samples where 0 < γ < 1 is the discount factor and K is the number of iterations. This result builds on the assumption

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

“…4. We give a concrete form of the perturbation response by distribution mismatch (Lemma 2) and show that when the assumptions on concentration coefficients in the existing literature [18,10,17,28] are satisfied or the eigenvalue decay of the kernel is fast, ∆ M (ǫ) decays fast with respect to ǫ (Proposition 5 and Lemma 3).…”

“…Based on the type of used assumptions, these works can be divided into two categories. The first category of upper bound [13,43,44] depends on the eigenvalue decay of kernel while the second category [19,28] requires accessibility to reference distributions that can uniformly bound all possible state-action distributions under admissible policies (called assumption on concentration coefficients). In this work, we show that the perturbational complexity ∆ M (ǫ) decays fast in both situations and establish an upper bound for the fitted reward algorithm (see Algorithm 2 in Section 5.1) and the fitted Q-iteration algorithm (see Algorithm 3 in Section 5.2) under the assumption that ∆ M (ǫ) decays fast.…”

“…Moreover, the kernel function approximation is closely related to neural network approximation, as established in the theory of neural tangent kernel [21] and Barron space [16]. RL with kernel function approximation has been recently studied in [18,13,43,44,28]. Still, results therein either suffer from the curse of dimensionality or require stringent assumptions on the kernel or the dynamics.…”

“…The optimal policy can be derived from the optimal Q-value function through the greedy policy. In practice, given any probability distribution ν, under certain conditions, one can estimate the optimal Q-value function Q * h in the sense of L 2 (ν) using Q-learning algorithm [9] or fitted Q-iteration algorithm (see Algorithm 3 or [17,28]). In other words, one can obtain Q * h , a ν-perturbation of Q * h .…”

“…If Π consists of all probability distributions, then the above semi-norm is just the L ∞ -norm, which is used to handle the distribution mismatch in the tabular and linear RL problems. However, for many common RKHSs, the L ∞ -estimation may suffer from the curse of dimensionality; see [26] and [28,Section 5] for a detailed discussion. The challenge of L ∞ -estimation in high dimensional space reveals the difficulty of RL problems in the RKHS compared to the tabular setting or linear function approximation.…”