Revisiting Approximate Linear Programming: Constraint-Violation Learning with Applications to Inventory Control and Energy Storage

Abstract

Approximate linear programs (ALPs) are well-known models for computing value function approximations (VFAs) of intractable Markov decision processes (MDPs). VFAs from ALPs have desirable theoretical properties, define an operating policy, and provide a lower bound on the optimal policy cost. However, solving ALPs near-optimally remains challenging, for example, when approximating MDPs with nonlinear cost functions and transition dynamics or when rich basis functions are required to obtain a good VFA. We address this tension between theory and solvability by proposing a convex saddle-point reformulation of an ALP that includes as primal and dual variables, respectively, a vector of basis function weights and a constraint violation density function over the state-action space. To solve this reformulation, we develop a proximal stochastic mirror descent (PSMD) method that learns regions of high ALP constraint violation via its dual update. We establish that PSMD returns a near-optimal ALP solution and a lower bound on the optimal policy cost in a finite number of iterations with high probability. We numerically compare PSMD with several benchmarks on inventory control and energy storage applications. We find that the PSMD lower bound is tighter than a perfect information bound. In contrast, the constraint-sampling approach to solve ALPs may not provide a lower bound, and applying row generation to tackle ALPs is not computationally viable. PSMD policies outperform problem-specific heuristics and are comparable or better than the policies obtained using constraint sampling. Overall, our ALP reformulation and solution approach broadens the applicability of approximate linear programming. This paper was accepted by Yinyu Ye, optimization.

Self-guided Approximate Linear Programs

“…Moreover, the FGLP policy optimality gap is at most 5% across these instances and improves by up to 8% the previously known gaps from the ALP in Lin et al (2019).…”

“…Third, we showcase how specific classes of random basis functions can be embedded in FALP and FGLP so that these linear programs can be solved using a constraint-generation technique and an extended version of a constraint sampling method. The latter method combines the popular constraint sampling approach in De Farias and Van Roy (2004) and a recent lower bounding technique from Lin et al (2019), and would be of independent interest for solving ALPs with fixed basis functions. Finally, we have made publicly available Python code implementing FALP and FGLP as well as benchmark methods.…”

“…This material discusses how ideas in Lin et al (2019) Table 1 since the perishable inventory control cost function is Lipschitz with constant L c > 0. In fact, it is easy to verify that L c = 2(γ L c oā + c hā + c b s + c dā + c lā ) and consequently, L y = (4 β 1 +Lc) /1−γ.…”

“…The solution of ALP given a fixed set of basis functions in Step (ii) is challenging since it has a large number of constraints and has been a topic of active research. It can be approached, for example, using techniques such as constraint generation, constraint sampling, and constraint-violation learning (see Lin et al 2019 for a recent overview of ALP solution techniques). The initial selection and potential modification of basis functions in steps (i) and (iv), respectively, are implementation bottlenecks when using ALP but this issue has received limited attention in the literature (Klabjan and Adelman 2007, Adelman and Klabjan 2012, and Bhat et al 2012.…”

“…Our experiments employ the sixteen instances from Lin et al (2019) and consider as a benchmark the ALP in this paper that embeds basis functions tailored to the application. We find that the FALP policy cost fluctuates significantly, that is, the policy cost can worsen as the iterations of the procedure shown in Figure 1(b) progress.…”