Controlled random sequences: methods of convex analysis and problems with functional constraints

Piunovskii, A. B.

doi:10.1070/rm1998v053n06abeh000090

Cited by 25 publications

(22 citation statements)

References 105 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Certain measurability, compactness and convexity properties of strategic measures for single decision maker problems were studied in [24], [41], [27] and [14]. In the following, we discuss the case for team problems, study some convexity properties and implications on the existence and structure of optimal policies.…”

Section: Nonclassical Casementioning

confidence: 99%

“…For a class of teams which are convex, one can reduce the search space to a smaller parametric class of policies, as discussed earlier. Finally, the strategic measure approach for single-decision maker problems and fully observed Markov models has been studied in [45] and [41], among other contributions in the literature.…”

Section: 2mentioning

confidence: 99%

“…For stochastic control problems, strategic measures are defined (see [45], [41], [24] and [27]) as the set of probability measures induced on the product spaces of the state and action pairs by measurable control policies: Given an initial distribution on the state, and a policy, one can uniquely define a probability measure on the product space. Certain measurability, compactness and convexity properties of strategic measures for single decision maker problems were studied in [24], [41], [27] and [14].…”

Section: Nonclassical Casementioning

confidence: 99%

See 2 more Smart Citations

Convex Analysis in Decentralized Stochastic Control, Strategic Measures, and Optimal Solutions

Yüksel¹,

Saldi²

2017

SIAM J. Control Optim.

View full text Add to dashboard Cite

Abstract. This paper is concerned with the properties of the sets of strategic measures induced by admissible team policies in decentralized stochastic control and the convexity properties in dynamic team problems. To facilitate a convex analytical approach, strategic measures for team problems are introduced. Properties such as convexity, compactness and Borel measurability under weak convergence topology are studied, and sufficient conditions for each of these properties are presented. These lead to existence of and structural results for optimal policies. It will be shown that the set of strategic measures for teams which are not classical is in general non-convex, but the extreme points of a relaxed set consist of deterministic team policies, which lead to their optimality for a given team problem under an expected cost criterion. Externally provided independent common randomness for static teams or private randomness for dynamic teams do not improve the team performance. The problem of when a sequential team problem is convex is studied and necessary and sufficient conditions for problems which include teams with a non-classical information structure are presented. Implications of this analysis in identifying probability and information structure dependent convexity properties are presented.Key words. Stochastic control, decentralized control, optimal control, convex analysis.AMS subject classifications. 93E03, 90B99, 49J551. Introduction. Team decision theory has its roots in control theory and economics. Marschak [37] was perhaps the first to introduce the basic elements of teams, and to provide the first steps toward the development of a team theory. Radner [42] provided foundational results for static teams, establishing connections between person-by-person optimality, stationarity, and team-optimality [38]. Contributions of Witsenhausen [56,57,58,54,53] on dynamic teams and characterization of information structures have been crucial in the progress of our understanding of dynamic teams. We refer the reader to Section 1.1, where Witsenhausen's intrinsic model, and characterization of information structures are discussed in detail. Further discussion on design of information structures in the context of team theory is available in [5,48,61].Convexity is a very important property for optimization problems. A property related to convex analysis that is relevant in team problems is the characterization of the sets of strategic measures; these are the probability measures induced on the exogenous variables, and measurement and action spaces by admissible control policies. In the context of single decision maker control problems, such measures have been studied extensively in [45,41,24,27]. A study of strategic measures for team problems has not been made to our knowledge, and it will be observed in this paper that many of the properties that are natural for fully-observed single-decision-maker stochastic control problems, such as convexity, do not generally extend to a large class of stochastic team problem...

show abstract

Section: Nonclassical Casementioning

confidence: 99%

Section: 2mentioning

confidence: 99%

Section: Nonclassical Casementioning

confidence: 99%

See 1 more Smart Citation

Convex Analysis in Decentralized Stochastic Control, Strategic Measures, and Optimal Solutions

Yüksel¹,

Saldi²

2017

SIAM J. Control Optim.

View full text Add to dashboard Cite

show abstract

“…The convex-analytical approach was introduced by Derman; see Derman [12] and references therein. Most of the later developments are recapped in monographs by Kallenberg [35], Borkar [6], Piunovskiy [42], Altman [1], and Hernández-Lerma and Lasserre [32], and in surveys by Piunovskiy [43] and Borkar [7]. It has various applications including to the Hamiltonian cycle problem; see Feinberg [21], Filar [28, § §3.3, 3.4], and Ejov et al [15].…”

mentioning

confidence: 99%

“…The conclusions of (i)-splitting on the entire space-has been established by Borkar [6, p. 37] for countable state discounted MDPs with compact action sets and by Piunovskiy [43,Theorem 6] for Borel state discounted MDPs that satisfy certain compactness and weak-continuity conditions. Related results for nonstationary policies appear in Feinberg [16,20].…”

mentioning

confidence: 99%

Splitting Randomized Stationary Policies in Total-Reward Markov Decision Processes

Feinberg

Rothblum

2012

Mathematics of OR

View full text Add to dashboard Cite

This paper studies a discrete-time total-reward Markov decision process (MDP) with a given initial state distribution. A (randomized) stationary policy can be split on a given set of states if the occupancy measure of this policy can be expressed as a convex combination of the occupancy measures of stationary policies, each selecting deterministic actions on the given set and coinciding with the original stationary policy outside of this set. For a stationary policy, necessary and sufficient conditions are provided for splitting it at a single state as well as sufficient conditions for splitting it on the whole state space. These results are applied to constrained MDPs. The results are refined for absorbing (including discounted) MDPs with finite state and actions spaces. In particular, this paper provides an efficient algorithm that presents the occupancy measure of a given policy as a convex combination of the occupancy measures of finitely many (stationary) deterministic policies. This algorithm generates the splitting policies in a way that each pair of consecutive policies differs at exactly one state. The results are applied to constrained problems to efficiently compute an optimal policy by computing and splitting a stationary optimal policy.Key words: Markov decision processes; occupancy measures; splitting occupancy measures; constrained Markov decision processes MSC2000 subject classification: Primary: 90C40; secondary: 97K60, 60J20, 60J22 OR/MS subject classification: Primary: dynamic programming/optimal control; secondary: deterministic Markov, finite state, infinite state History: Received April 25, 2008; revised December 11, 2010, and October 16, 2011. Published online in Articles in Advance January 9, 2012.1. Introduction. This paper is concerned with a discrete-time Markov decision process (MDP) with a given distribution of an initial state and with total-reward criteria. It investigates whether and how a stationary policy can be replaced by another policy that is defined as a random selection among policies that are deterministic on a prescribed set of states and coincide with the original stationary policy outside of that set. Contributions are presented for MDPs with finite state and actions sets, for MDPs with countable state sets, and for MDPs with Borel state and action spaces.An MDP is said to be absorbing if its expected lifetime is finite under every policy. In particular, a discounted MDP can be presented as an absorbing MDP. For an absorbing MDP with a fixed initial state distribution, the occupancy measure of a policy specifies the expectation of the number of visits to each measurable set of state-action pairs. The expected total reward for the policy can be expressed as the integral of the one-step reward function with respect to the policy's occupancy measure. Thus, optimizing the expected total rewards is reduced to optimizing a linear function over the set of occupancy measures. This is the basic idea of the convexanalytic approach which provides useful methods for solving MDPs w...

show abstract

Approximate Optimal Cost and Policies of First Passage Markov Decision Processes with Countable-State Space and Discount Factors

Tang

2021

Proceedings of IncoME-V &Amp; CEPE Net-2020

View full text Add to dashboard Cite

Controlled random sequences: methods of convex analysis and problems with functional constraints

Cited by 25 publications

References 105 publications

Convex Analysis in Decentralized Stochastic Control, Strategic Measures, and Optimal Solutions

Convex Analysis in Decentralized Stochastic Control, Strategic Measures, and Optimal Solutions

Splitting Randomized Stationary Policies in Total-Reward Markov Decision Processes

Approximate Optimal Cost and Policies of First Passage Markov Decision Processes with Countable-State Space and Discount Factors

Contact Info

Product

Resources

About