Fast LSTD Using Stochastic Approximation: Finite Time Analysis and Application to Traffic Control

Abstract

We propose a stochastic approximation based method with randomisation of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our method results in an O(d) improvement in complexity in comparison to regular LSTD, where d is the dimension of the data. We provide convergence rate results for our proposed method, both in high probability and in expectation. Moreover, we also establish that using our scheme in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. This result coupled with the low complexity of our method makes it attractive for implementation in big data settings, where d is large. Further, we also analyse a similar low-complexity alternative for least squares regression and provide finite-time bounds there. We demonstrate the practicality of our method for LSTD empirically by combining it with the LSPI algorithm in a traffic signal control application.Several machine learning problems involve solving a linear system of equations from a given set of training data. In this paper we consider the problem of policy evaluation in reinforcement learning (RL) using the method of temporal differences (TD). Given a fixed training data set, one popular temporal difference algorithm for policy evaluation is LSTD Bradtke and Barto [1996]. However, LSTD is computationally expensive as it requires O(d 2 ) computations. We propose a stochastic approximation (SA) based algorithm that draws data samples from a uniform distribution on the training set. From the finite time analyses that we provide, we observe our algorithm converges at the optimal rate, in high probability as well as in expectation. Moreover, using our scheme in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. This finding coupled with the significant decrease in the computational cost of our algorithm, makes it appealing in the canonical big data settings.The problem considered here is to estimate the value function V π of a given policy π. Temporal difference (TD) methods are well-known in this context, and they are known to converge to the fixed point, where T π is the Bellman operator (see Section 2.1 for a precise definition). A popular approach to overcome the curse of dimensionality associated with large state spaces is to parameterize the value function using a linear function approximation architecture. For every s in the state space S, we approximate V π (s) ≈ θ T φ(s), where φ(·) is a d-dimensional feature vector with d << |S|, and θ is a tunable parameter. The function approximation variant of TD Tsitsiklis and Van Roy [1997] is known to converge to the fixed point of Φθ = ΠT π (Φθ), where Π is the orthogonal projection onto the space within which we approximate the value function, and Φ is the feature matrix that characterises this space.LSTD estimates the fixed point of ΠT π using empirical data D := {(s i , r i , s i ), i = 1, . . . , T )} obtained by simulating t...

A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation

“…Analysis of related algorithms: A number of papers analyze algorithms related to and inspired by the classic TD algorithm. First, among others, Antos et al [2008], , , Pires and Szepesvári [2012], Prashanth et al [2013] and Tu and Recht [2018] analyze least-squares temporal difference learning (LSTD). Yu and Bertsekas [2009] study the related least-squares policy iteration algorithm.…”

Concentration bounds for temporal difference learning with linear function approximation: the case of batch data and uniform sampling

“…This paper is an extended version of an earlier work (see Prashanth et al 2014). This work corrects the errors in the earlier work by using significant deviations in the proofs, and includes additional simulation experiments.…”

“…This work corrects the errors in the earlier work by using significant deviations in the proofs, and includes additional simulation experiments. Finally, by Narayanan and Szepesvári (2017), the authors list a few problems with the results and proofs in the conference version (Prashanth et al 2014), and the corrections incorporated in this work address the comments by Narayanan and Szepesvári (2017).…”

Finite-Sample Analysis of Proximal Gradient TD Algorithms

“…The analysis is a bit different than the one in Lazaric et al [2010b] and the bound is weaker in terms of d and ν. Another recent result is by Prashanth et al [2014] that use stochastic approximation to solve LSTD(0), where the resulting algorithm is exactly TD(0) with random sampling (samples are drawn i.i.d. and not from a trajectory), and report a Markov design bound (the bound is computed only at the states used by the algorithm) of O( d nν ) for LSTD(0).…”