Learning More Complex Actions with Deep Reinforcement Learning

Wang, Chenxi; Du, Youtian; Xie, Shengyuan; Lu, Yongdi

doi:10.1109/irc52146.2021.00028

Cited by 1 publication

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the majority of existing theoretical analysis of RL is only applicable to the tabular setting (see, e.g., [1][2][3][4][5][6]), in which both the state and action spaces are discrete and finite, and no function approximation is involved. Relatively simple function approximation methods, such as the linear model in [7,8] or generalized linear model in [9,10], have been recently studied in the context of RL with various statistical estimates. Yet, these results are not sufficient to explain the practical success of RL algorithms in high dimensions.…”

Section: Introductionmentioning

confidence: 99%

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Long¹,

Han²

2022

JML

View full text Add to dashboard Cite

Most existing theoretical analysis of reinforcement learning (RL) is limited to the tabular setting or linear models due to the difficulty in dealing with function approximation in high dimensional space with an uncertain environment. This work offers a fresh perspective into this challenge by analyzing RL in a general reproducing kernel Hilbert space (RKHS). We consider a family of Markov decision processes M of which the reward functions lie in the unit ball of an RKHS and transition probabilities lie in a given arbitrary set. We define a quantity called perturbational complexity by distribution mismatch ∆ M (ǫ) to characterize the complexity of the admissible state-action distribution space in response to a perturbation in the RKHS with scale ǫ. We show that ∆ M (ǫ) gives both the lower bound of the error of all possible algorithms and the upper bound of two specific algorithms (fitted reward and fitted Q-iteration) for the RL problem. Hence, the decay of ∆ M (ǫ) with respect to ǫ measures the difficulty of the RL problem on M. We further provide some concrete examples and discuss whether ∆ M (ǫ) decays fast or not in these examples. As a byproduct, we show that when the reward functions lie in a high dimensional RKHS, even if the transition probability is known and the action space is finite, it is still possible for RL problems to suffer from the curse of dimensionality.

show abstract