Robustness to Approximations and Model Learning in MDPs and POMDPs

Kara, Ali; Yüksel, Serdar

doi:10.1007/978-3-030-76928-4_9

Cited by 4 publications

(4 citation statements)

References 91 publications

(183 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the constructed kernel P is weakly continuous, the cost function is bounded continuous, and the state space is compact, the proof is then a corollary of [53,Theorem 4.27] (see also [41,Theorem 11]) by viewing the finite window truncation as a quantization (with a uniformly bounded radius for each bin) of the state space under the product topology.…”

Section: Discounted Cost Criterion: Refined Existence Results and Nearmentioning

confidence: 99%

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Yüksel¹

2023

Preprint

View full text Add to dashboard Cite

We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

show abstract

Section: Discounted Cost Criterion: Refined Existence Results and Nearmentioning

confidence: 99%

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Yüksel¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Models and Methods. Classical discrete-time Markov decision processes (MDPs) are considered in [3,4,6,8,9,12,15]; continuous-time Markov, semi-Markov, and more general processes are considered in [2,5,10,11,19]. Chapters [4,14,18] are about various types of stochastic games, including the game against the nature [4].…”

Section: Introductionmentioning

confidence: 99%

“…As for the methods, dynamic programming is useful on many occasions [3,4,6,[8][9][10]15]. When some probabilities (e.g., describing the dynamics of the process) are not precisely known, the Bayesian approach [9,12,14], Q-learning [3,4], optimal filtering [19], robust control [1,4,12], and H 2 control [5] can be useful. Let us also mention variational inequalities [11] and self-organizing algorithms [7].…”

Section: Introductionmentioning

confidence: 99%

“…Let us also mention variational inequalities [11] and self-organizing algorithms [7]. Many authors suggested new effective numerical methods for tackling optimal control problems [3,4,6,7,10,12,13], especially arising from real-life case studies. Results of essential computer calculations and simulations are presented in [1, 3-5, 7, 9, 10, 13, 14, 18, 20].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Modern Trends in Controlled Stochastic Processes:

2021

Emergence, Complexity and Computation

View full text Add to dashboard Cite

The Emergence, Complexity and Computation (ECC) series publishes new developments, advancements and selected topics in the fields of complexity, computation and emergence. The series focuses on all aspects of reality-based computation approaches from an interdisciplinary point of view especially from applied sciences, biology, physics, or chemistry. It presents new ideas and interdisciplinary insight on the mutual intersection of subareas of computation, complexity and emergence and its impact and limits to any computing based on physical limits (thermodynamic and quantum limits, Bremermann's limit, Seth Lloyd limits…) as well as algorithmic limits (Gödel's proof and its impact on calculation, algorithmic complexity, the Chaitin's Omega number and Kolmogorov complexity, non-traditional calculations like Turing machine process and its consequences,…) and limitations arising in artificial intelligence. The topics are (but not limited to) membrane computing, DNA computing, immune computing, quantum computing, swarm computing, analogic computing, chaos computing and computing on the edge of chaos, computational aspects of dynamics of complex systems (systems with self-organization, multiagent systems, cellular automata, artificial life,…), emergence of complex systems and its computational aspects, and agent based computation. The main aim of this series is to discuss the above mentioned topics from an interdisciplinary point of view and present new ideas coming from mutual intersection of classical as well as modern methods of computation. Within the scope of the series are monographs, lecture notes, selected contributions from specialized conferences and workshops, special contribution from international experts.

show abstract

Reinforcement Learning for Fully Observable Models: Convergence to Near Optimality under Weak Feller Continuity

Yüksel,

Başar

2024

Systems &Amp; Control: Foundations &Amp; Applications

View full text Add to dashboard Cite

Robustness to Approximations and Model Learning in MDPs and POMDPs

Abstract: This thesis develops a topological approach to robustness to incorrect-incomplete models in MDPs and POMDPs, and applications of this study to approximations, model learning and reinforcement learning.

Cited by 4 publications

References 91 publications

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Modern Trends in Controlled Stochastic Processes:

Reinforcement Learning for Fully Observable Models: Convergence to Near Optimality under Weak Feller Continuity

Contact Info

Product

Resources

About