Learning Rates for Q-Learning

Even-Dar, Eyal; Mansour, Yishay

doi:10.1007/3-540-44581-1_39

Cited by 229 publications

(277 citation statements)

References 4 publications

Supporting

Mentioning

261

Contrasting

Unclassified

Order By: Relevance

“…Bertsekas and Tsitsiklis have verified the convergence of stochastic iterative algorithms [3] when (41) holds. In fact many traditional RL algorithms have been proved to be stochastic iterative algorithms [3], [4], [47] and QRL is the same as traditional RL, and main differences lie in:…”

Section: A Convergence Of Qrlmentioning

confidence: 99%

Quantum Reinforcement Learning

Dong

Chen

2005

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract-The key approaches for machine learning, especially learning in unknown probabilistic environments are new representations and computation mechanisms. In this paper, a novel quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state superposition principle and quantum parallelism, a framework of value updating algorithm is introduced. The state (action) in traditional RL is identified as the eigen state (eigen action) in QRL. The state (action) set can be represented with a quantum superposition state and the eigen state (eigen action) can be obtained by randomly observing the simulated quantum state according to the collapse postulate of quantum measurement. The probability of the eigen action is determined by the probability amplitude, which is parallelly updated according to rewards. Some related characteristics of QRL such as convergence, optimality and balancing between exploration and exploitation are also analyzed, which shows that this approach makes a good tradeoff between exploration and exploitation using the probability amplitude and can speed up learning through the quantum parallelism. To evaluate the performance and practicability of QRL, several simulated experiments are given and the results demonstrate the effectiveness and superiority of QRL algorithm for some complex problems. The present work is also an effective exploration on the application of quantum computation to artificial intelligence.Index Terms-quantum reinforcement learning, state superposition, collapse, probability amplitude, Grover iteration.

show abstract

Section: A Convergence Of Qrlmentioning

confidence: 99%

Quantum Reinforcement Learning

Dong

Chen

2005

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Interest in Linear programming (LP) for solving MDPs has been renewed (Defarias and van Roy, 2003) because of the well-known stability properties of LP solvers. Other areas that are attracting interest are: risk-sensitive RL (Borkar, 2002;Geibel and Wysotzki, 2005), factored MDPs (Schuurmans and Patrascu, 2002), and analyzing computational complexity (Evan-Dar and Mansour, 2003).…”

Section: Discussionmentioning

confidence: 99%

Reinforcement Learning: A Tutorial Survey and Recent Advances

Gosavi

2009

INFORMS Journal on Computing

258

View full text Add to dashboard Cite

In the last few years, Reinforcement Learning (RL), also called adaptive (or approximate) dynamic programming (ADP), has emerged as a powerful tool for solving complex sequential decision-making problems in control theory. Although seminal research in this area was performed in the artificial intelligence (AI) community, more recently, it has attracted the attention of optimization theorists because of several noteworthy success stories from operations management. It is on large-scale and complex problems of dynamic optimization, in particular the Markov decision problem (MDP) and its variants, that the power of RL becomes more obvious. It has been known for many years that on large-scale MDPs, the curse of dimensionality and the curse of modeling render classical dynamic programming (DP) ineffective. The excitement in RL stems from its direct attack on these curses, allowing it to solve problems that were considered intractable, via classical DP, in the past. The success of RL is due to its strong mathematical roots in the principles of DP, Monte Carlo simulation, function approximation, and AI. Topics treated in some detail in this survey are: Temporal differences, Q-Learning, semi-MDPs and stochastic games. Several recent advances in RL, e.g., policy gradients and hierarchical RL, are covered along with references. Pointers to numerous examples of applications are provided. This overview is aimed at uncovering the mathematical roots of this science, so that readers gain a clear understanding of the core concepts and are able to use them in their own research. The survey points to more than 100 references from the literature.

show abstract

“…The sequence (Q t ; t ≥ 0) is indeed known to converge to Q * when appropriate local learning rates are used (Tsitsiklis, 1994;Jaakkola et al, 1994). 15 The rate of convergence of Q-learning was studied by in an asymptotic setting and later by Even-Dar and Mansour (2003) in a finite-sample setting. The key observation that lead to the discovery of Q-learning is that unlike the optimal state values, the optimal action-values can be expressed as expectations (compare Equations (13) and (15)).…”

Section: Q-learning In Finite Mdpsmentioning

confidence: 99%