An analysis of temporal-difference learning with function approximation

Abstract

We discuss the temporal-difference learning algorithm, as applied to approximating the cost-to-go function of an infinite-horizon discounted Markov chain. The algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic Markov chain with a finite or infinite state space. We present a proof of convergence (with probability one), a characterization of the limit of convergence, and a bound on the resulting approximation error. Furthermore, our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal-difference learning.In addition to proving new and stronger positive results than those previously available, we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators. First, we prove that divergence may occur when updates are not based on trajectories of the Markov chain. This fact reconciles positive and negative results that have been discussed in the literature, regarding the soundness of temporal-difference learning. Second, we present an example illustrating the possibility of divergence when temporaldifference learning is used in the presence of a nonlinear function approximator.

Efficient Solution Algorithms for Factored MDPs

“…One approach is to approximate the solution using an approximate value function with a compact representation. A common choice is the use of linear value functions as an approximation -value functions that are a linear combination of potentially non-linear basis functions (Bellman, Kalaba, & Kotkin, 1963;Sutton, 1988;Tsitsiklis & Van Roy, 1996b). Our work builds on the ideas of Parr (1999, 2000), by using factored (linear) value functions, where each basis function is restricted to some small subset of the domain variables.…”

“…An approach along the lines described above has been used in various papers, with several recent theoretical and algorithmic results (Schweitzer & Seidmann, 1985;Tsitsiklis & Van Roy, 1996b;Van Roy, 1998;Koller & Parr, 1999. However, these approaches suffer from a problem that we might call "norm incompatibility."…”

Recursive Least-Squares Learning with Eligibility Traces

“…When the behavior policy π 0 is different from the target policy π, a sufficient condition for contraction is that λ be close enough to 1; indeed, when λ tends to 1, the spectral radius of T λ tends to zero and can potentially balance an expansion of the projection Π 0 . In the off-policy case, when γ is sufficiently big, a small value of λ can make Π 0 T λ expansive (see [20] for an example in the case λ = 0) and off-policy LSPE(λ) will then diverge. Eventually, Equations (5) and (8) show that when λ = 1, both LSTD(λ) and LSPE(λ) asymptotically coincide (as T 1 V does not depend on V ).…”

“…Eventually, we consider 2 experiments involving an MDP and a projection due to [20], in order to illustrate possible numerical issues when solving the projected fixed-point Eq. (5).…”

“…TD learning with eligibility traces [19], known as TD(λ), provide a nice bridge between both approaches, and by controlling the bias/variance trade-off [12], their use can significantly speed up learning. When the value function is approximated through a linear architecture, the depth λ of the eligibility traces is also known to control the quality of approximation [20]. Overall, the use of these traces often plays an important practical role.…”