Optimal measures and Markov transition kernels

Belavkin, Roman V.

doi:10.1007/s10898-012-9851-1

Cited by 17 publications

(16 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That is, the criterion is defined as a conditional extremum of expected utility under a relative entropy constraint. It therefore satisfies all of the axioms for expected utility and hence describes means of decision making under uncertainty that is independent of any specific definition of information [35]. It also does not, as noted by Belavkin [36], suffer from some of the perceived flaws of utility theory [37,38].…”

Section: Introductionmentioning

confidence: 83%

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Sledge

Prı́ncipe

2019

IEEE Trans. Games

View full text Add to dashboard Cite

Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity.We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcementlearning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.Index Terms-Value of information, constrained search, reinforcement learning, information theory Isaac J. Sledge is with the 2 objective of the agent is to clear the environment of pellets while navigating around the ghosts. However, after activating certain power-ups, the ghosts become vulnerable for a brief period of time. The agent can consume these ghosts for a score boost.The switch in ghost dynamics necessitates a change in the game-play strategy, since multiple distinct modes of behavior are required under different conditions [4,5]. Despite the need for multi-modal behaviors, conventional reinforcement-learning approaches have focused on constructing monolithic policies. Such policies would implement the same agent behaviors regardless of the vulnerability of the ghosts. Although it is possible to represent multimodal behavior with these policies, it can be difficult to learn such behavior. This is, in part, due to risk. For instance, throughout the learning process, an agent may have learned to avoid colliding with the ghosts. Without straying from this behavior, the agent will not learn that there are instances where it can safely chase the ghosts.In this paper, we consider an information-theoretic learning [6] approach for performing constrained stochastic searches that promote a continuum of risk-averse to risk-favoring agent behaviors during reinforcement learning. This, in turn, leads to a principled exploration of the state-action space that aids in the...

show abstract

Section: Introductionmentioning

confidence: 83%

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Sledge

Prı́ncipe

2019

IEEE Trans. Games

View full text Add to dashboard Cite

show abstract

“…Mathematically speaking, crossover is a mixing process with extensive local search in a subspace [2,3]. This can be seen by an example.…”

Section: Evolutionary Operatorsmentioning

confidence: 99%

Analysis of Algorithms

Yang¹

2014

Nature-Inspired Optimization Algorithms

View full text Add to dashboard Cite

“…The author finds it more convenient to work in the category of linear spaces and making the restriction to an affine subspace when necessary. Thus, we shall assume axioms (1), (2) and (3). Substituting z = −x − y into (2) gives also…”

Section: Choice Under Uncertaintymentioning

confidence: 99%

“…This observation was illustrated on a 2-simplex in [15], and it clearly shows why the expected utility theory alone cannot explain the switch from risk-averse to risk-taking behaviour observed in many examples discussed above. Thus, it appears that human decision-makers violate the linear axioms (1) and (2), and several 'non-expected' utility theories have been proposed, such as the regret theory [14] (see [22,16,17] for a review of many others).…”

Section: Why Is This a Paradox?mentioning

confidence: 99%

“…Remarkably, the value of information function has two distinct branches -one is concave, representing the upper frontier of expected utility, while another is convex, representing the lower frontier of expected utility. Interestingly, it was shown recently that these geometric properties do not depend on the definition of information itself, but follow from linearity of the expected utility [2]. In this section, we discuss the classical notion of value of information, its generalization, and how it can be related to asymmetry of risk.…”

Section: Risk and Value Of Informationmentioning

confidence: 99%

See 1 more Smart Citation

Asymmetry of Risk and Value of Information

Belavkin

2014

Springer Proceedings in Mathematics &Amp; Statistics

Self Cite

View full text Add to dashboard Cite

The von Neumann and Morgenstern theory postulates that rational choice under uncertainty is equivalent to maximization of expected utility (EU). This view is mathematically appealing and natural because of the affine structure of the space of probability measures. Behavioural economists and psychologists, on the other hand, have demonstrated that humans consistently violate the EU postulate by switching from risk-averse to risk-taking behaviour. This paradox has led to the development of descriptive theories of decisions, such as the celebrated prospect theory, which uses an $S$-shaped value function with concave and convex branches explaining the observed asymmetry. Although successful in modelling human behaviour, these theories appear to contradict the natural set of axioms behind the EU postulate. Here we show that the observed asymmetry in behaviour can be explained if, apart from utilities of the outcomes, rational agents also value information communicated by random events. We review the main ideas of the classical value of information theory and its generalizations. Then we prove that the value of information is an $S$-shaped function, and that its asymmetry does not depend on how the concept of information is defined, but follows only from linearity of the expected utility. Thus, unlike many descriptive and `non-expected' utility theories that abandon the linearity (i.e. the `independence' axiom), we formulate a rigorous argument that the von Neumann and Morgenstern rational agents should be both risk-averse and risk-taking if they are not indifferent to information.Comment: 16 pages, 2 figure

show abstract

Optimal measures and Markov transition kernels

Cited by 17 publications

References 32 publications

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Analysis of Algorithms

Asymmetry of Risk and Value of Information

Contact Info

Product

Resources

About