Average Optimality in Dynamic Programming with General State Space

Schäl, Manfred

doi:10.1287/moor.18.1.163

Cited by 169 publications

(218 citation statements)

References 15 publications

Supporting

Mentioning

216

Contrasting

Order By: Relevance

“…For unbounded costs, optimality inequalities are used instead of optimality equations to construct a stationary average cost optimal policy. Cavazos-Cadena [7] provided an example, when optimality inequalities hold while optimality equations do not.Schäl [26] developed a theory for Borel state spaces and compact action sets. Two types of continuity assumptions for transition probabilities are considered in Schäl [26]: the setwise and weak continuity.…”

mentioning

confidence: 99%

“…Two types of continuity assumptions for transition probabilities are considered in Schäl [26]: the setwise and weak continuity. For a countable 592 Mathematics of Operations Research 37(4), pp.…”

mentioning

confidence: 99%

“…Schäl's [26] assumptions on compactness of action sets and lower semicontinuity of cost functions in the action argument are replaced in Hernández-Lerma [18] by a more general assumption; namely, that the cost functions are inf-compact in the action argument. For weakly continuous transition probabilities and possibly noncompact action sets, Feinberg and Lewis [14] proved the existence of stationary optimal policies for MDPs with cost functions being inf-compact in both state and action arguments when, in addition to Schäl's [26] boundedness assumption on the relative discounted value at each state, the so-called local boundedness condition was assumed.…”

mentioning

confidence: 99%

“…According to Schäl [26,Proposition 1.3], under Assumption (G), if there exists a measurable function u → 0 + and a stationary policy such that…”

mentioning

confidence: 99%

“…Also, denote by S the set of nonempty subsets of . A set-valued mapping F → S is upper semicontinuous at x ∈ if, for any neighborhood G of the set F x , there is a neighborhood of x; say, U x such that F y ⊆ G for all y ∈ U x (see, e.g., Berge For weakly continuous transition probabilities, the following basic assumptions were considered in Schäl [26].…”

mentioning

confidence: 99%

See 4 more Smart Citations

Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

2012

View full text Add to dashboard Cite

This paper presents sufficient conditions for the existence of stationary optimal policies for average cost Markov decision processes with Borel state and action sets and weakly continuous transition probabilities. The one-step cost functions may be unbounded, and the action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average cost optimal actions by discount optimal actions.Key words: Markov decision process; average cost per unit time; optimality inequality; optimal policy MSC2000 subject classification: Primary: 90C40; secondary: 90C39 OR/MS subject classification: Primary: dynamic programming/optimal control; secondary: Markov, infinite state 1. Introduction. This paper provides sufficient conditions for the existence of stationary optimal policies for average cost Markov decision processes (MDPs) with Borel state and action sets and weakly continuous transition probabilities. The cost functions may be unbounded, and the action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions (Theorems 1, 3, and 4), (ii) a new sufficient condition for average cost optimality based on optimality inequalities (Theorem 2), and (iii) approximations of average cost optimal actions by discount optimal actions (Theorem 5).For infinite-horizon MDPs, there are two major criteria: average costs per unit time and expected total discounted costs. The former is typically more difficult to analyze. The so-called vanishing discount factor approach is often used to approximate average costs per unit time by normalized expected total discounted costs. The literature on average cost MDPs is vast. Most of the earlier results are surveyed in Arapostathis et al. [1]. Here, we mention just a few references.For finite-state and action sets, Derman [10] proved the existence of stationary average cost optimal policies. This result follows from Blackwell [6] and it also was independently proved by Viskov and Shiryaev [31]. When either the state set or action set is infinite, even -optimal policies may not exist for some > 0; Ross [25], Dynkin and Yushkevich [11, Chapter 7], Feinberg [12, §5]. For a finite-state set and compact action sets, optimal policies may not exist; Bather [2], Chitashvili [9], and Dynkin and Yushkevich [11, Chapter 7].For MDPs with finite-state and action sets, there exist stationary policies satisfying optimality equations (see Dynkin and Yushkevich [11, Chapter 7], where these equations are called canonical), and, furthermore, any stationary policy satisfy...

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

“…According to Schäl [26,Proposition 1.3], under Assumption (G), if there exists a measurable function u → 0 + and a stationary policy such that…”

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

2012

View full text Add to dashboard Cite

show abstract

Bibliography

1998

Stochastic Dynamic Programming and the Control of Queueing Systems

View full text Add to dashboard Cite

A review on solar forecasting and power management approaches for energy‐harvesting wireless sensor networks

Kakkar

2020

Int J Communication

View full text Add to dashboard Cite

Summary For rechargeable wireless sensor nodes, effective power management is of prime importance because of the stochastic behaviour of the environmental resources. A key issue in integrating solar resources with wireless sensor networks (WSNs) is the need of precise irradiance measurements and power to resource modelling. WSNs are employed in an adhoc manner comprises of numerous sensing nodes and organised as a network for the sake of checking and balancing the environmental factors. Each node has sensing, computation, communication, and locomotion capabilities but operates with limited battery life. Energy harvesting is a way of powering these WSNs by harvesting energy from the environment. By considering harvested energy as an energy source, certain considerations are different from that of battery‐operated networks. Nondeterministic energy availability with respect to time is the reason behind these differences, which put a limit on the maximum rate at which energy can be used. Thus, reliable knowledge of solar radiation is essential for informed design, deployment planning, and optimal management of energy in rechargeable WSNs. Further, power management is essential in self‐powerssed networks to efficiently utilize the available energy. In this paper, a detailed survey on different solar forecasting techniques has been presented for precise energy estimates. A detailed study on energy efficient power management techniques is also proposed to address the feasibility of energy‐harvesting approach in WSNs.

show abstract

Average Optimality in Dynamic Programming with General State Space

Cited by 169 publications

References 15 publications

Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Bibliography

A review on solar forecasting and power management approaches for energy‐harvesting wireless sensor networks

Contact Info

Product

Resources

About