Weakly Coupled Dynamic Program: Information and Lagrangian Relaxations

Ye, Fan; Zhu, Helin; Zhou, Enlu

doi:10.1109/tac.2017.2731817

Cited by 15 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Independently, Ye et al (2014) show a similar decoupling for weakly coupled DPs and Lagrangian relaxations; like Hawkins (2003) and Adelman and Mersereau (2008), they assume a product form for state transition probabilities that does not hold in the multiclass queueing application we study in Section 5. Ye et al (2014) study the use of subgradient methods to solve their decoupled inner problems and provide a "gap analysis" that describes a limit on how much slack can be introduced by using Lagrangian relaxations of the inner problems. In our multiclass queueing example with grouped Lagrangian relaxations, using a subgradient method with groups of size 4 would still require solving DPs with 10,000τ states, and the runtimes could still be substantial, particuarly given that subgradient methods can be slow to converge.…”

Section: Resultsmentioning

confidence: 92%

“…Desai et al (2011) show how to improve on bounds from approximate linear programming with perfect information relaxations, and Smith (2011, 2014) show that using information relaxations with "gradient penalties" improves bounds from relaxed DP models when the DP has a convex structure. In independent work, Ye et al (2014) study weakly coupled DPs and show that improved bounds can be obtained by combining perfect information relaxations with Lagrangian relaxations; they do not consider reformulations. We also use Lagrangian relaxations in our multiclass queueing examples and find the reformulations to be quite useful in that application.…”

Section: Literature Review and Outlinementioning

confidence: 99%

See 1 more Smart Citation

Information Relaxation Bounds for Infinite Horizon Markov Decision Processes

Brown

Haugh

2017

Operations Research

View full text Add to dashboard Cite

We consider the information relaxation approach for calculating performance bounds for stochastic dynamic programs (DPs), following Brown et al. [Brown DB, Smith JE, Sun P (2010) Information relaxations and duality in stochastic dynamic programs. Oper. Res. 58(4, Part 1):785-801]. This approach generates performance bounds by solving problems with relaxed nonanticipativity constraints and a penalty that punishes violations of these constraints. In this paper, we study infinite horizon DPs with discounted costs and consider applying information relaxations to reformulations of the DP. These reformulations use different state transition functions and correct for the change in state transition probabilities by multiplying by likelihood ratio factors. These reformulations can greatly simplify solutions of the information relaxations, both in leading to finite horizon subproblems and by reducing the number of states that need to be considered in these subproblems. We show that any reformulation leads to a lower bound on the optimal cost of the DP when used with an information relaxation and a penalty built from a broad class of approximate value functions. We refer to this class of approximate value functions as subsolutions, and this includes approximate value functions based on Lagrangian relaxations as well as those based on approximate linear programs. We show that the information relaxation approach, in theory, recovers a tight lower bound using any reformulation and is guaranteed to improve on the lower bounds from subsolutions. Finally, we apply information relaxations to an inventory control application with an autoregressive demand process, as well as dynamic service allocation in a multiclass queue. In our examples, we find that the information relaxation lower bounds are easy to calculate and are very close to the expected cost using simple heuristic policies, thereby showing that these heuristic policies are nearly optimal. Keywords: infinite horizon dynamic programs • information relaxations • Lagrangian relaxations • inventory control • multiclass queues

show abstract

Section: Resultsmentioning

confidence: 92%

Section: Literature Review and Outlinementioning

confidence: 99%

Information Relaxation Bounds for Infinite Horizon Markov Decision Processes

Brown

Haugh

2017

Operations Research

View full text Add to dashboard Cite

show abstract

“…A relevant literature review is given by Chand et al (2002), and an early theoretical treatment is offered by Sethi and Sorger (1991). Finally, our overall approach can also be used in other applications that involve decision making over an infinite horizon, such as the multiclass queueing and inventory control problems discussed in Brown and Haugh (2017) or dynamic product promotion (Ye et al, 2018).…”

Section: Production and Operations Managementmentioning

confidence: 99%

Finite‐horizon approximate linear programs for capacity allocation over a rolling horizon

Vossen

You

Zhang

2022

Production and Operations Management

View full text Add to dashboard Cite

Approximate linear programs (ALPs) have been used extensively to approximately solve stochastic dynamic programs that suffer from the well‐known curse of dimensionality. Due to canonical results establishing the optimality of stationary value functions and policies for infinite‐horizon dynamic programs, the literature has largely focused on approximation architectures that are stationary over time. In a departure from this literature, we apply a nonstationary approximation architecture to an infinite‐dimensional linear programming formulation of the stochastic dynamic programs. We solve the resulting problems using a finite‐horizon approximation. Such finite‐horizon approximations are common in the theoretical analysis of infinite‐horizon linear programs, but have not been considered in the approximate linear programming literature. We illustrate the approach on a rolling‐horizon capacity allocation problem using an affine approximation architecture. We obtain three main results. First, nonstationary approximations can substantially improve upper bounds on the optimal revenue. Second, the upper bounds from the finite‐horizon approximation monotonically decrease as the horizon length increases, and converge to the upper bound from the infinite‐horizon approximation. Finally, the improvement does not come at the expense of tractability, as the resulting ALPs admit compact representations and can be solved efficiently. The resulting approximations also produce strong heuristic policies and significantly reduce optimality gaps in numerical experiments.

show abstract

“…Geoffrion's theorem [26] ensures that we can compute z LR by solving the Dantzig-Wolfe reformulation of MILP (21) within a column generation approach. The proof of Proposition 8 and the full details of the column generation algorithm are available in Appendix E. Note that Lagrangian relaxation has already been used in the literature on weakly coupled stochastic dynamic programs to compute upper bounds [2,30,61].…”

Section: A Tractable Upper Bound Through Lagrangian Relaxationmentioning

confidence: 99%

“…Such models have been applied to stochastic inventory routing with limited vehicle capacity [32], stochastic multi-product dispatch problems [46], scheduling problems [60], resources allocation [27], revenue management [56] among others [30]. The specific structure of these problems can then be leveraged in mathematical programming formulations that use approximate value functions [17] or approximate moments [9] as variables, notably using the Lagrangian relaxation of non-anticipativity constraints [11] or of the linking constraints [61]. All the mathematical programs in this paper can be formulated either using moments variables (or marginal probabilities) or value function variables.…”

Section: Introductionmentioning

confidence: 99%

Integer programming for weakly coupled stochastic dynamic programs with partial information

Cohen¹,

Parmentier²

2020

Preprint

View full text Add to dashboard Cite

This paper introduces algorithms for problems where a decision maker has to control a system composed of several components and has access to only partial information on the state of each component. Such problems are difficult because of the partial observations, and because of the curse of dimensionality that appears when the number of components increases. Partially observable Markov decision processes (POMDPs) have been introduced to deal with the first challenge, while weakly coupled stochastic dynamic programs address the second. Drawing from these two branches of the literature, we introduce the notion of weakly coupled POMDPs. The objective is to find a policy maximizing the total expected reward over a finite horizon. Our algorithms rely on two ingredients. The first, which can be used independently, is a mixed integer linear formulation for generic POMDPs that computes an optimal memoryless policy. The formulation is strengthened with valid cuts based on a probabilistic interpretation of the dependence between random variables, and its linear relaxation provide a practically tight upper bound on the value of an optimal history-dependent policies. The second is a collection of mathematical programming formulations and algorithms which provide tractable policies and upper bounds for weakly coupled POMDPs. Lagrangian relaxations, fluid approximations, and almost sure constraints relaxations enable to break the curse of dimensionality. We test our generic POMDPs formulations on benchmark instance forms the literature, and our weakly coupled POMDP algorithms on a maintenance problem. Numerical experiments show the efficiency of our approach.

show abstract

Weakly Coupled Dynamic Program: Information and Lagrangian Relaxations

Cited by 15 publications

References 29 publications

Information Relaxation Bounds for Infinite Horizon Markov Decision Processes

Information Relaxation Bounds for Infinite Horizon Markov Decision Processes

Finite‐horizon approximate linear programs for capacity allocation over a rolling horizon

Integer programming for weakly coupled stochastic dynamic programs with partial information

Contact Info

Product

Resources

About