On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Jaakkola, Tommi S.; Jordan, Michael I.; Singh, Shveta

doi:10.21236/ada276517

Cited by 152 publications

(132 citation statements)

References 5 publications

Supporting

Mentioning

128

Contrasting

Unclassified

Order By: Relevance

“…Before stating our convergence theorem, we must introduce the following standard assumption concerning the stepsize sequence: A proof of Theorem 1 is provided in Appendix C. We prove the theorem by showing that the algorithm corresponds to a stochastic approximation involving a maximum norm contraction, and then appeal to a theorem concerning asynchronous stochastic approximation due to Tsitsiklis (1994) (see also (Jaakola, Jordan, and Singh, 1994)), which is discussed in Appendix B, and a theorem concerning multi-representation contractions presented and proven in Appendix A.…”

Section: Convergence Theoremmentioning

confidence: 99%

“…We then have the following result (Tsitsiklis, 1994) (related results are obtained in (Jaakola, Jordan, and Singh, 1994)): THEOREM 4 Let Assumption 6 and Assumption 1 of Section 5 on the stepsizes ai(t) hold and suppose that the mapping T is a contraction with respect to the maximum norm. Then, V(t) converges to the unique fixed point V* ofT, with probability 1.…”

Section: Assumption 6 (A) For Ever) I and T We Have E[rh(t ) [ -)C(mentioning

confidence: 99%

See 1 more Smart Citation

Feature-based methods for large scale dynamic programming

1996

View full text Add to dashboard Cite

Abstract. We develop a methodological framework and present a few different ways in which dynamic programming and compact representations can be combined to solve large scale stochastic control problems. In particular, we develop algorithms that employ two types of feature-based compact representations; that is, representations that involve feature extraction and a relatively simple approximation architecture. We prove the convergence of these algorithms and provide bounds on the approximation error. As an example, one of these algorithms is used to generate a strategy for the game of Tetris. Furthermore, we provide a counterexample illustrating the difficulties of integrating compact representations with dynamic programming, which exemplifies the shortcomings of certain simple approaches.

show abstract

Section: Convergence Theoremmentioning

confidence: 99%

Section: Assumption 6 (A) For Ever) I and T We Have E[rh(t ) [ -)C(mentioning

confidence: 99%

Feature-based methods for large scale dynamic programming

1996

View full text Add to dashboard Cite

show abstract

“…Tal algoritmoé baseado nos conceitos do método de diferenças temporais e sua convergência para os valoresótimos de Q (Q * (s, a)) independe da política que está sendo utilizada. A expressão de atualização do valor de Q do algoritmo Q-Learningé a seguinte: O Q-Learning foi o primeiro método de aprendizagem por reforço a possuir fortes provas de convergência [28].É uma técnica muito simples que calcula diretamente as ações sem avaliações intermediárias e sem uso de modelo. Watkins mostrou que se cada para estado-ação for visitado um número infinito de vezes e com um valor de α adequado, a função de valor Q irá convergir com probabilidade 1 para Q * .…”

Section: Q-learningunclassified

Uma Contribuição a Solução do Problema dos k-Servos Usando Aprendizagem por Reforço

Júnior¹,

de²

2016

Anais Do 7. Congresso Brasileiro De Redes Neurais

View full text Add to dashboard Cite

“…The convergence is generally established with probability 1 (w.p.1) because of the RM basis of RL. The first results for asynchronous convergence of discounted Q-Learning (Jaakola et al, 1994;Tsitsiklis, 1994) were based on norm contractions. The idea of ordinary differential equations (ODEs) for proving convergence under asynchronous conditions was proposed in Borkar (1998), where it was shown that the iterate tracks an ODE, which is much slower than that shown to exist under synchronous conditions (Kushner and Clark, 1978).…”

Section: Semi-markov Decision Problemsmentioning

confidence: 99%

Reinforcement Learning: A Tutorial Survey and Recent Advances

Gosavi

2009

INFORMS Journal on Computing

258

View full text Add to dashboard Cite

In the last few years, Reinforcement Learning (RL), also called adaptive (or approximate) dynamic programming (ADP), has emerged as a powerful tool for solving complex sequential decision-making problems in control theory. Although seminal research in this area was performed in the artificial intelligence (AI) community, more recently, it has attracted the attention of optimization theorists because of several noteworthy success stories from operations management. It is on large-scale and complex problems of dynamic optimization, in particular the Markov decision problem (MDP) and its variants, that the power of RL becomes more obvious. It has been known for many years that on large-scale MDPs, the curse of dimensionality and the curse of modeling render classical dynamic programming (DP) ineffective. The excitement in RL stems from its direct attack on these curses, allowing it to solve problems that were considered intractable, via classical DP, in the past. The success of RL is due to its strong mathematical roots in the principles of DP, Monte Carlo simulation, function approximation, and AI. Topics treated in some detail in this survey are: Temporal differences, Q-Learning, semi-MDPs and stochastic games. Several recent advances in RL, e.g., policy gradients and hierarchical RL, are covered along with references. Pointers to numerous examples of applications are provided. This overview is aimed at uncovering the mathematical roots of this science, so that readers gain a clear understanding of the core concepts and are able to use them in their own research. The survey points to more than 100 references from the literature.

show abstract

On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Cited by 152 publications

References 5 publications

Feature-based methods for large scale dynamic programming

Feature-based methods for large scale dynamic programming

Uma Contribuição a Solução do Problema dos k-Servos Usando Aprendizagem por Reforço

Reinforcement Learning: A Tutorial Survey and Recent Advances

Contact Info

Product

Resources

About