Control of Markov Chains With Safety Bounds

Arapostathis, Ari; Kumar, Ratnesh; Hsu, Shun‐Pin

doi:10.1109/tase.2005.853392

Cited by 21 publications

(13 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper continues our prior work ( [3], [2]) on controlled Markov chains with safety constraints. We generalize the constraint set to be any linear convex set and also propose a method to compute a long-run average optimal, safe and stationary control policy.…”

Section: B Two-state Markov Chainssupporting

confidence: 65%

“…In this case, the necessary and sufficient condition for a policy to be safety enforcing can be seen in (Theorem 3.1, [3]). Example 4.6: Suppose in Example 3.6 the given constraint set is Π c = {π ∈ Π | EC ≤ (1 + 10%)C * , π(i) ≤ (1 + 10%)π * (i), i = 1, 2, 3} (8) where the expected cost EC := 3 i=1 π i C i .…”

Section: A Searching Algorithmmentioning

confidence: 99%

“…Previous work with this setting can be found in [2], [3], [10], where the constraint was specified by the type of upper/lower bound on the system's state probability distribution vector. In this paper we consider more general constraints in the form of convex polyhedral sets and we call these the constraint sets.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On optimal control of Markov chains with safety constraint

Hsu

Arapostathis²,

Kumar

2006

2006 American Control Conference

Self Cite

View full text Add to dashboard Cite

Abstract-We study the control of completely observed Markov chains with safety bounds as introduced in [3], but with more general safety constraints and the added requirement of optimality. In [3], the safety bounds were specified as unitinterval valued vector pairs (lower and upper bounds for each component of the state probability distribution). In this paper we generalize the constraint set to be any linear convex set and present a way to compute a stationary control policy which is safe (i.e., maintains the safety of the distribution that is initially safe) and at the same time it is long-run average optimal. We propose a linear programming formulation for computing such a safe optimal policy. Under a simplifying assumption that the optimal policy is ergodic, we present a finitely-terminating iterative algorithm to compute the maximal invariant safe set (MISS) where the initial distribution must lie so that the future distributions always remain safe. Our approach allows us to calculate an upper bound for the number of iterations needed for the algorithm to terminate. In particular, for the two-state chains we show that at most one iteration is needed to compute the MISS.

show abstract

Section: B Two-state Markov Chainssupporting

confidence: 65%

Section: A Searching Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

On optimal control of Markov chains with safety constraint

Hsu

Arapostathis²,

Kumar

2006

2006 American Control Conference

Self Cite

View full text Add to dashboard Cite

show abstract

“…First, the set of state feedback controllers that satisfy the requirement for any given safe initial state probability distribution is identified, and, then, the set of all safe initial state probability distributions for a given state feedback controller is found. The work of (Arapostathis et al, 2005) further extends the results of (Arapostathis et al, 2003) to a more general class of Markov chains. Also, a safety requirement is given by two vectors representing lower and upper bounds on the state probability vector.…”

Section: Related Workmentioning

confidence: 56%

“…Also, a safety requirement is given by two vectors representing lower and upper bounds on the state probability vector. Also, (Arapostathis et al, 2005) presents a more general iterative algorithm to find safe initial distributions, and provides the number of iterations needed for the result to be reached.…”

Section: Related Workmentioning

confidence: 99%

Probabilistic Supervisory Control of Probabilistic Discrete Event Systems

Pantelic

Postma

Lawford

2009

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

This thesis considers probabilistic supervisory control of probabilistic discrete event systems (PDES). PDES are modeled as generators of probabilistic languages. The probabilistic supervisors employed are a generalization of the deterministic ones previously employed in the literature. At any state, the supervisor enables/disables events with certain probabilities. The probabilistic supervisory control problem (PSCP) that has previously been considered in the literature is revisited: find, if possible, a supervisor under whose control the behavior of a plant is identical to a given probabilistic specification. The existing results are unified, complemented with a solution of a special case and the computational analysis of synthesis problem and the solution.The central place in the thesis is given to the solution of the optimal probabilistic supervisory control problem (OPSCP) in the framework: if the conditions for the existence of probabilistic supervisor for PSCP problem are not satisfied, find a probabilistic supervisor such that the achievable behaviour is as close as possible to the desired behaviour. The proximity is measured using the concept of pseudometric on states of generators. The distance between two systems is defined as the distance in the pseudometric between the initial states of the corresponding generators.The pseudometric is adopted from the research in formal methods community and is defined as the greatest fixed point of a monotone function. Starting from this definition, we suggest two algorithms for finding the distances in the pseudometric. Further, we give a logical characterization of the same pseudometric such that the distance between two systems is measured by a formula that distinguishes between the systems the most. A trace characterization of the pseudometric is then derived from the logical characterization by which the pseudometric measures the difference of (appropriately discounted) iv probabilities of traces and sets of traces generated by systems, as well as some more complicated properties of traces. Then, the solution to the optimal probabilistic supervisory control problem is presented.Further, the solution of the problem of approximation of a given probabilistic generator with another generator of a prespecified structure is suggested such that the new model is as close as possible to the original one in the pseudometric (probabilistic model fitting). The significance of the approximation is then discussed. While other applications are briefly discussed, a special attention is given to the use of ideas of probabilistic model fitting in the solution of a modified optimal probabilistic supervisory control problem.v

show abstract

Control of continuous‐time Markov chains with safety constraints

Hsu

2012

Intl J Robust & Nonlinear

View full text Add to dashboard Cite

In this work the controlled continuous-time finite-state Markov chain with safety constraints is studied. The constraints are expressed as a finite number of inequalities, whose intersection forms a polyhedron. A probability distribution vector is called safe if it is in the polyhedron. Under the assumptions that the controlled Markov chain is completely observable and the controller induces a unique stationary distribution in the interior of the polyhedron, the author identifies the supreme invariant safety set (SISS) where a set is called an invariant safety set if any probability distribution in the set is initially safe and remains safe as time evolves. In particular, the necessary and sufficient condition for the SISS to be the polyhedron itself is given via linear programming formulations. A closed-form expression for the condition is also derived as the constraints impose only upper and/or lower bounds on the components of the distribution vectors. If the condition is not satisfied, a finite time bound is identified and used to characterize the SISS. Numerical examples are provided to illustrate the results. Copyright (c) 2011 John Wiley & Sons, Ltd

show abstract

Control of Markov Chains With Safety Bounds

Cited by 21 publications

References 8 publications

On optimal control of Markov chains with safety constraint

On optimal control of Markov chains with safety constraint

Probabilistic Supervisory Control of Probabilistic Discrete Event Systems

Control of continuous‐time Markov chains with safety constraints

Contact Info

Product

Resources

About