A version of Geiringer‐like theorem for decision making in the environments with randomness and incomplete information

Abstract

Received (?? ?? 2011) Revised (Day Month Year)Purpose-In recent years Monte-Carlo sampling methods, such as Monte Carlo tree search, have achieved tremendous success in model free reinforcement learning. A combination of the so called upper confidence bounds policy to preserve the "exploration vs. exploitation" balance to select actions for sample evaluations together with massive computing power to store and to update dynamically a rather large pre-evaluated game tree lead to the development of software that has beaten the top human player in the game of Go on a 9 by 9 board. Much effort in the current research is devoted to widening the range of applicability of the Monte-Carlo sampling methodology to partially observable Markov decision processes with non-immediate payoffs. The main challenge introduced by randomness and incomplete information is to deal with the action evaluation at the chance nodes due to drastic differences in the possible payoffs the same action could lead to. The aim of this article is to establish a version of a theorem that originated from population genetics and has been later adopted in evolutionary computation theory that will lead to novel Monte-Carlo sampling algorithms that provably increase the AI potential. Due to space limitations the actual algorithms themselves will be presented in the sequel papers, however, the current paper provides a solid mathematical foundation for the development of such algorithms and explains why they are so promising. Design/Methodology/Approach-In the current paper we set up a mathematical framework, state and prove a version of a Geiringer-like theorem that is very well-suited for the development of Mote-Carlo sampling algorithms to cope with randomness and incomplete information to make decisions. From the framework it will be clear that such algorithm increase what seems like a limited sample of rollouts exponentially in size by exploiting the symmetry within the state space at little or no additional * EPSRC EP/D003/05/1 "Amorphous Computing" and EPSRC EP/I009809/1 "Evolutionary Approximation Algorithms for Optimization: Algorithm Design and Complexity Analysis" Grants. computational cost. Appropriate notions of recombination (or crossover) and schemata are introduced to stay inline with the traditional evolutionary computation terminology. The main theorem is proved using the methodology developed in the PhD thesis of the first author, however the general case of nonhomologous recombination presents additional challenges that have been overcome thanks to a lovely application of the classical and elementary tool known as the "Markov inequality" together with the lumping quotients of Markov chains techniques developed and successfully applied by the authors in the previous research for different purposes. This methodology will be mildly extended to establish the main result of the current article. In addition to establishing the Geiringer-like theorem for Monte Carlo sampling, which is the central objective of this paper, we also strengthen the...

A further generalization of the finite-population geiringer-like theorem for pomdps to allow recombination overarbitrary set covers

“…Just as in [3], a convenient way to represent a similarity relation C on a set of states S is to assign a positive integer to each similarity set O ∈ C in a one-to-one fashion. Each element of a set O labeled by an integer l is then uniquely determined by an additional alphabet symbol.…”

“…Each element of a set O labeled by an integer l is then uniquely determined by an additional alphabet symbol. Unlike the case in [3], it is possible for the same state to be labeled in a number of different ways as long as the corresponding integer labels differ. An example appears below.…”

“…Just as in definition 2.4 we will assume that s i k = s j q whenever i = j (which, in accordance with definition 2.4, is as strong as requiring that s i k = s j q whenever i = j or k = q) 3 Moreover, we also assume that the terminal labels f i are also all distinct within the same population, i.e. for i = j the terminal labels f i = f j 4 In a very special case when…”

“…5 We now continue with the rigorous definitions of crossover. Crossover or recombination operators will be defined in terms of the similarities induced by the set cover much in the same way as it has been done in [3]. Nonetheless, here we are not limited to a single equivalence relation on the set of states, and many more recombination operators will be introduced depending on various similarity sets in C. For this reason it is convenient to introduce the following notion: For every recombination compatible triple we can introduce the following two recombination operators: Definition 2.7.…”

“…The central idea behind results such as the main theorem of this article (a special case where the similarity relation is an equivalence relation (i.e. C is a partition of S) has been established in [3]) is that one can actually anticipate the long term (or limiting) frequency of occurrence of various rollouts provided such a genetic programming routine has ran. This type of predictions is what the Geiringer-like theorems are about.…”