Structural Properties of Stochastic Dynamic Programs

Smith, James E.; McCardle, Kevin F.

doi:10.1287/opre.50.5.796.365

Cited by 95 publications

(79 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3. See also Hopenhayn and Prescott (1992), Topkis (1998), and Smith and McCardle (2002) for other conditions that yield monotonicity of optimal solutions to dynamic programs. 4.…”

Section: Proofmentioning

confidence: 99%

Mean Field Equilibrium in Dynamic Games with Strategic Complementarities

Adlakha

Johari

2013

Operations Research

View full text Add to dashboard Cite

We study a class of stochastic dynamic games that exhibit strategic complementarities between players; formally, in the games we consider, the payoff of a player has increasing differences between her own state and the empirical distribution of the states of other players. Such games can be used to model a diverse set of applications, including network security models, recommender systems, and dynamic search in markets. Stochastic games are generally difficult to analyze, and these difficulties are only exacerbated when the number of players is large (as might be the case in the preceding examples).We consider an approximation methodology called mean field equilibrium to study these games. In such an equilibrium, each player reacts to only the long-run average state of other players. We find necessary conditions for the existence of a mean field equilibrium in such games. Furthermore, as a simple consequence of this existence theorem, we obtain several natural monotonicity properties. We show that there exist a "largest" and a "smallest" equilibrium among all those where the equilibrium strategy used by a player is nondecreasing, and we also show that players converge to each of these equilibria via natural myopic learning dynamics; as we argue, these dynamics are more reasonable than the standard best-response dynamics. We also provide sensitivity results, where we quantify how the equilibria of such games move in response to changes in parameters of the game (for example, the introduction of incentives to players).

show abstract

“…3. See also Hopenhayn and Prescott (1992), Topkis (1998), and Smith and McCardle (2002) for other conditions that yield monotonicity of optimal solutions to dynamic programs. 4.…”

Section: Proofmentioning

confidence: 99%

Mean Field Equilibrium in Dynamic Games with Strategic Complementarities

Adlakha

Johari

2013

Operations Research

View full text Add to dashboard Cite

show abstract

“…For examples of such results, see Benjaafar and ElHafsi (2006), ElHafsi et al (2008), ElHafsi (2009), and Benjaafar et al (2011. See also Smith and McCardle (2002) for sufficient conditions ensuring convexity in a multivariate Markovian inventory model. However, the existence of counterexamples proves that convexity need not hold for our model (see Nadar et al 2014).…”

Section: Introductionmentioning

confidence: 99%

Technical Note—Optimal Structural Results for Assemble-to-Order Generalized M-Systems

Nadar

Akan

Scheller‐Wolf

2014

Operations Research

View full text Add to dashboard Cite

We consider an assemble-to-order generalized M-system with multiple components and multiple products, batch ordering of components, random lead times, and lost sales. We model the system as an infinite-horizon Markov decision process and seek an optimal policy that specifies when a batch of components should be produced (i.e., inventory replenishment) and whether an arriving demand for each product should be satisfied (i.e., inventory allocation). We characterize optimal inventory replenishment and allocation policies under a mild condition on component batch sizes via a new type of policy: lattice-dependent base stock and lattice-dependent rationing.

show abstract

“…To propose the prototypical procedure of proving the existence of a monotonic optimal policy, we first define a P property as follows: We therefore propose an approach, similar to Proposition 5 in [18], as follows: (n) (x, a) = C(x, a) + β x ∈X P a xx V (n−1) (x ) has P property for all P property functions V (n−1) and n.…”

Section: Structured Properties Of Dynamic Programmingmentioning

confidence: 99%

“…In such a high dimensional MDP, the curse of dimensionality 1 becomes more evident [17]; the computation load grows quickly if the cardinality of any tuple in the state variable is large. To relieve the curse, one solution is to qualitatively understand the model and prove the existence of a monotonic optimal policy [18]. Then, a low complexity algorithm or a model-free learning method can be proposed, e.g., simultaneous perturbation stochastic approximation (SPSA) [19,20].…”

Section: Introductionmentioning

confidence: 99%

Structured optimal transmission control in network-coded two-way relay channels

Ding

Sadeghi

Kennedy

2015

J Wireless Com Network

View full text Add to dashboard Cite

This paper considers a transmission control problem in network-coded two-way relay channels (NC-TWRC), where the relay buffers randomly arrived packets from two users, and the channels are assumed to be fading. The problem is modeled by a discounted infinite horizon Markov decision process (MDP). The objective is to find an adaptive transmission control policy that minimizes the packet delay, buffer overflow, transmission power consumption and downlink error rate simultaneously and in the long run. By using the concepts of submodularity, multimodularity and L -convexity, we study the structure of the optimal policy searched by dynamic programming (DP) algorithm. We show that the optimal transmission policy is nondecreasing in queue occupancies and/or channel states under certain conditions such as the chosen values of parameters in the MDP model, channel modeling method, and the preservation of stochastic dominance in the transitions of system states. Based one these results, we propose to use two low-complexity algorithms for searching the optimal monotonic policy: monotonic policy iteration (MPI) and discrete simultaneous perturbation stochastic approximation (DSPSA). We show that MPI reduces the time complexity of DP, and DSPSA is able to adaptively track the optimal policy when the statistics of the packet arrival processes change with time.

show abstract

Structural Properties of Stochastic Dynamic Programs

Cited by 95 publications

References 16 publications

Mean Field Equilibrium in Dynamic Games with Strategic Complementarities

Mean Field Equilibrium in Dynamic Games with Strategic Complementarities

Technical Note—Optimal Structural Results for Assemble-to-Order Generalized M-Systems

Structured optimal transmission control in network-coded two-way relay channels

Contact Info

Product

Resources

About