On incorporating the paradigms of discretization and Bayesian estimation to create a new family of pursuit learning automata

Zhang, Xuan; Granmo, Ole‐Christoffer; Oommen, B. John

doi:10.1007/s10489-013-0424-x

Cited by 34 publications

(24 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During the last decade, SE ri has been considered as the state-of-art algorithm for a long time, however, some recently proposed algorithms [8], [9] claim a faster convergence than SE ri . To make a comprehensive comparison among currently available techniques, as well as to verify the effectiveness of the proposed parameter-free scheme, in this section, PFLA is compared with several classic parameter-based learning automata schemes, including DP ri [20], DGPA [4], DBPA [6], DGCPA * [8], SE ri [5], GBSE [9] and LELA R [7].…”

Section: Simulation Resultsmentioning

confidence: 99%

“…In [6], DBPA was proposed where the posterior distribution of estimatedĉ i is represented by a beta distribution Beta(α, β), the parameter α and β record the number of times that a specific action has been rewarded and penalized respectively. Then the 95 th percentile of the cumulative posterior distribution is utilized as estimation of c i .…”

Section: ) a Comprehensive Comparison Among Recently Proposedmentioning

confidence: 99%

“…At the beginning, as we know nothing about the actions, a non-informative (uniform) prior distribution is advised to infer the posterior distribution. So α i and β i should be set identically to 1, exactly the same as in [6], [23].…”

Section: Initialization Of Beta Distributionsmentioning

confidence: 99%

See 2 more Smart Citations

A Parameter-Free Gradient Bayesian Two-Action Learning Automaton Scheme

Yan

et al. 2016

Proceedings of the 2015 International Conference on Communications, Signal Processing, and Systems

View full text Add to dashboard Cite

Reinforcement learning is one of the subjects of Artificial Intelligence and learning automata have been considered as one of the most powerful tools in this research area. A learning automaton (LA) is a learning machine that can learn an optimal action through interacting with stochastic environments. However, its performance highly depends on the selection of a configurable learning speed. Granmo proposed a Bayesian Learning Automaton (BLA), which is reported to not rely on such external parameters, to solve Two-Armed Bernoulli Bandit problem in 2010. In this paper, we demonstrate the BLA algorithm from a learning automata perspective. Furthermore, we devote efforts to improving the convergence speed of BLA by incorporating a gradient descent-like method and then proposed a novel Gradient Bayesian Learning Automaton (GBLA). It has been demonstrated by extensive simulations that GBLA is faster than BLA and traditional algorithms such as DP RI , and outperforms the state-of-art SE RI algorithm as well.

show abstract

Section: Simulation Resultsmentioning

confidence: 99%

Section: ) a Comprehensive Comparison Among Recently Proposedmentioning

confidence: 99%

See 1 more Smart Citation

A Parameter-Free Gradient Bayesian Two-Action Learning Automaton Scheme

Yan

et al. 2016

Proceedings of the 2015 International Conference on Communications, Signal Processing, and Systems

View full text Add to dashboard Cite

show abstract

“…Table 3 shows the comparison between the t 0 calculated from (18) and the average number of iterations needed for the LA to converge, in practice. As the CPA and ACPA are well-established algorithms, we know that numerous experiments have been conducted to confirm their validity, and so we merely use the figures from [30] to record the practical results.…”

Section: Remarkmentioning

confidence: 99%

A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions

et al. 2014

Self Cite

View full text Add to dashboard Cite

The most difficult part in the design and analysis of Learning Automata (LA) consists of the formal proofs of their convergence accuracies. The mathematical techniques used for the different families (Fixed Structure, Variable Structure, Discretized etc.) are quite distinct. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and within this family, the set of Pursuit algorithms have been considered to be the pioneering schemes. Informally, if the environment is stationary, their ε-optimality is defined as their ability to converge to the optimal action with an arbitrarily large probability, if the learning parameter is sufficiently small/large. The existing proofs of all the reported EAs follow the same fundamental principles, and to clarify this, in the interest of simplicity, we shall concentrate on the family of Pursuit algorithms. Recently, it has been reported Ryan and Omkar (J Appl Probab 49(3): [795][796][797][798][799][800][801][802][803][804][805] 2012) that the previous proofs for ε-optimality of all the reported EAs have a common flaw.The flaw lies in the condition which apparently supports the so-called "monotonicity" property of the probability of selecting the optimal action, which states that after some time instant t 0 , the reward probability estimates will be ordered correctly forever. The authors of the various proofs have rather offered a proof for the fact that the reward probability estimates are ordered correctly at a single point of time after t 0 , which, in turn, does not guarantee the ordering forever, rendering the previous proofs incorrect. While in Ryan and Omkar (J Appl Probab 49(3):795-805, 2012), a rectified proof was presented to prove the ε-optimality of the Continuous Pursuit Algorithm (CPA), which was the pioneering EA, in this paper, a new proof is provided for the Absorbing CPA (ACPA), i.e., an algorithm which follows the CPA paradigm but which artificially has absorbing states whenever any action probability is arbitrarily close to unity. Unlike the previous flawed proofs, instead of examining the monotonicity property of the action probabilities, it rather examines their submartingale property, and then, unlike the traditional approach, invokes the theory of Regular functions to prove that the probability of converging to the optimal action can be made arbitrarily close to unity. We believe that the proof is both unique and pioneering, and adds insights into the convergence of different EAs. It can also form the basis for formally demonstrating the ε-optimality of other Estimator algorithms which are artificially rendered absorbing.

show abstract

“…In contrast to any LA scheme presented in the literature, our solution is especially designed for deterministic environments. & Our current solution extends the family of pursuit LA algorithms [3,38,39] to solve deterministic optimization problems. A common feature for all legacy pursuit algorithms is to estimate the reward probability of each action and pursue the action with the highest reward.…”

Section: Introductionmentioning

confidence: 99%

A team of pursuit learning automata for solving deterministic optimization problems

2020

View full text Add to dashboard Cite

Learning Automata (LA) is a popular decision-making mechanism to "determine the optimal action out of a set of allowable actions" [1]. The distinguishing characteristic of automata-based learning is that the search for an optimal parameter (or decision) is conducted in the space of probability distributions defined over the parameter space, rather than in the parameter space itself [2]. In this paper, we propose a novel LA paradigm that can solve a large class of deterministic optimization problems. Although many LA algorithms have been devised in the literature, those LA schemes are not able to solve deterministic optimization problems as they suppose that the environment is stochastic. In this paper, our proposed scheme can be seen as the counterpart of the family of pursuit LA developed for stochastic environments [3]. While classical pursuit LAs can pursue the action with the highest reward estimate, our pursuit LA rather pursues the collection of actions that yield the highest performance by invoking a team of LA. The theoretical analysis of the pursuit scheme does not follow classical LA proofs, and can pave the way towards more schemes where LA can be applied to solve deterministic optimization problems. Furthermore, we analyze the scheme under both a constant learning parameter and a time-decaying learning parameter. We provide some experimental results that show how our Pursuit-LA scheme can be used to solve the Maximum Satisfiability (Max-SAT) problem. To avoid premature convergence and better explore the search space, we enhance our scheme with the concept of artificial barriers recently introduced in [4]. Interestingly, although our scheme is simple by design, we observe that it performs well compared to sophisticated state-of-the-art approaches.

show abstract

On incorporating the paradigms of discretization and Bayesian estimation to create a new family of pursuit learning automata

Cited by 34 publications

References 31 publications

A Parameter-Free Gradient Bayesian Two-Action Learning Automaton Scheme

A Parameter-Free Gradient Bayesian Two-Action Learning Automaton Scheme

A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions

A team of pursuit learning automata for solving deterministic optimization problems

Contact Info

Product

Resources

About