On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming

Abstract

In the linear programming approach to approximate dynamic programming, one tries to solve a certain linear program-the ALP-that has a relatively small number K of variables but an intractable number M of constraints. In this paper, we study a scheme that samples and imposes a subset of m M constraints. A natural question that arises in this context is: How must m scale with respect to K and M in order to ensure that the resulting approximation is almost as good as one given by exact solution of the ALP? We show that, given an idealized sampling distribution and appropriate constraints on the K variables, m can be chosen independently of M and need grow only as a polynomial in K. We interpret this result in a context involving controlled queueing networks. 1. Introduction. Due to the "curse of dimensionality," Markov decision processes typically have a prohibitively large number of states, rendering exact dynamic programming methods intractable and calling for the development of approximation techniques. This paper represents a step in the development of a linear programming approach to approximate dynamic programming (de Farias and Van Roy 2003;Schweitzer and Seidmann 1985; Zin 1993, 1997). This approach relies on solving a linear program that generally has few variables but an intractable number of constraints. In this paper, we propose and analyze a constraint sampling method for approximating the solution to this linear program. We begin in this section by discussing our working problem formulation, the linear programming approach, constraint sampling, results of our analysis, and related literature.

Approximate Dynamic Programming—II: Algorithms

“…Not surprisingly, this can produce very large linear programs very quickly. The linear programming method received a new lease on life with the work of de Farias and Van Roy [23], who introduced ADP concepts to this method. The first step to reduce complexity involves introducing basis functions to simplify the representation of the value function, giving us [23] then introduced the novel idea of using a Monte Carlo sample of the constraints.…”

“…The linear programming method received a new lease on life with the work of de Farias and Van Roy [23], who introduced ADP concepts to this method. The first step to reduce complexity involves introducing basis functions to simplify the representation of the value function, giving us [23] then introduced the novel idea of using a Monte Carlo sample of the constraints. As of this writing, this method is receiving considerable attention in the research community, but as with all ADP algorithms, considerable work is needed to produce robust algorithms that work in an industrial setting.…”

Approximate Dynamic Programming by Practical Examples

“…Finally, besides value iteration and policy iteration, we can also use the linear programming method. This method-that for MDPs suffers from the curse of dimensionality since we need a decision variable for each state, and a constraint for each state-action pair-receives attention in the ADP community due to the work of Farias and Roy (2004), who introduced ADP concepts to this method, incorporating value function approximations into the linear program and sampling of the constraints.…”

Relaxations of Approximate Linear Programs for the Real Option Management of Commodity Storage

“…Schweitzer and Seidmann (1985) introduce the approximate linear programming approach to approximate dynamic programming. de Farias and Van Roy (2001, 2006 analyze it. Applications of this approach include Trick and Zin (1997) in economics; Adelman (2004) and Adelman and Klabjan (2011) in inventory control; Adelman (2007), Farias andVan Roy (2007), and Zhang and Adelman (2009) in revenue management; and Morrison and Kumar (1999), Van Roy (2001, 2003), Moallemi et al (2008), and Veatch (2010) in queuing.…”