Non-ergodic Markov decision processes with a constraint on the asymptotic failure rate: general class of policies

Boussemart, Michel; Limnios, Nikolaos; Fillion, Jean-Marc

doi:10.1081/stm-120002780

Cited by 6 publications

(3 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both of these reinforcement learning methods assume that every Markov chain induced by a policy is irreducible, which allows only a single recurrent class as with ergodic and unichain assumptions described earlier. The Lagrangian approach has also been applied to specific stochastic policy linear programming formulations relevant to aircraft maintenance problems where the asymptotic failure is to be kept below some small threshold (Boussemart & Limnios, 2004;Boussemart et al, 2002).…”

Section: Related Workmentioning

confidence: 99%

“…Steady-state planning has applications in several areas, such as deriving maintenance plans for various systems, including aircraft maintenance, where the asymptotic failure rate of components must be kept below some small threshold (Boussemart & Limnios, 2004;Boussemart, Limnios, & Fillion, 2002). Optimal routing problems for communication networks have also been proposed in which data throughput must be maximized subject to constraints on average delay and packet drop metrics (Lazar, 1983).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Steady-State Planning in Expected Reward Multichain MDPs

Atia¹,

Beckus²,

Alkhouri³

et al. 2021

jair

View full text Add to dashboard Cite

The planning domain has experienced increased interest in the formal synthesis of decision-making policies. This formal synthesis typically entails finding a policy which satisfies formal specifications in the form of some well-defined logic. While many such logics have been proposed with varying degrees of expressiveness and complexity in their capacity to capture desirable agent behavior, their value is limited when deriving decision-making policies which satisfy certain types of asymptotic behavior in general system models. In particular, we are interested in specifying constraints on the steady-state behavior of an agent, which captures the proportion of time an agent spends in each state as it interacts for an indefinite period of time with its environment. This is sometimes called the average or expected behavior of the agent and the associated planning problem is faced with significant challenges unless strong restrictions are imposed on the underlying model in terms of the connectivity of its graph structure. In this paper, we explore this steady-state planning problem that consists of deriving a decision-making policy for an agent such that constraints on its steady-state behavior are satisfied. A linear programming solution for the general case of multichain Markov Decision Processes (MDPs) is proposed and we prove that optimal solutions to the proposed programs yield stationary policies with rigorous guarantees of behavior.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Steady-State Planning in Expected Reward Multichain MDPs

Atia¹,

Beckus²,

Alkhouri³

et al. 2021

jair

View full text Add to dashboard Cite

show abstract

“…A high level of local instability of the frequency causes a high probability of early/late deliveries that are both undesirable (early deliveries lead to wasting the items that are not consumed before expiration, and late deliveries lead to a shortage of items). • dependability, i.e., an upper bound on failure frequency (see, e.g., (Boussemart and Limnios 2004;Boussemart, Limnios, and Fillion 2002)). If this bound is locally violated with considerable probability, a user may interpret this as a violation of the dependability guarantee.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Local Satisfaction of Long-Run Average Objectives in Markov Decision Processes

Klaška,

Kučera,

Kůr

et al. 2024

AAAI

View full text Add to dashboard Cite

Long-run average optimization problems for Markov decision processes (MDPs) require constructing policies with optimal steady-state behavior, i.e., optimal limit frequency of visits to the states. However, such policies may suffer from local instability in the sense that the frequency of states visited in a bounded time horizon along a run differs significantly from the limit frequency. In this work, we propose an efficient algorithmic solution to this problem.

show abstract

Constrained Markov Decision Process for the Industry

BOUSSEMART,

LIMNIOS

2024

Computational Methods and Mathematical Modeling in Cyberphysics and Engineering Applications 1

View full text Add to dashboard Cite

Non-ergodic Markov decision processes with a constraint on the asymptotic failure rate: general class of policies

Cited by 6 publications

References 2 publications

Steady-State Planning in Expected Reward Multichain MDPs

Steady-State Planning in Expected Reward Multichain MDPs

Optimizing Local Satisfaction of Long-Run Average Objectives in Markov Decision Processes

Constrained Markov Decision Process for the Industry

Contact Info

Product

Resources

About