Discretized pursuit learning automata

Oommen, B. John; Lanctot, J. Kevin

doi:10.1109/21.105092

Cited by 147 publications

(141 citation statements)

References 16 publications

Supporting

Mentioning

140

Contrasting

Order By: Relevance

“…In other words, the proofs of these results are already found in the literature, namely in [26] and in [23] respectively. The proofs are thus not repeated here.…”

mentioning

confidence: 54%

“…The proofs for the convergence of PAs have been studied and reported for decades in [6], [23], [24], [25] [26], which, unfortunately, all have a common flaw that has been recently discovered by the authors of [27]. Further, the authors of [27] submitted a new proof for the convergence of the CPA which adequately rectified the flawed proofs.…”

Section: Proofs Of Pasmentioning

confidence: 98%

“…The first PA was designed to operate by updating the action probabilities based on the L R−I paradigm. Later, Oommen and Lanctot [23] presented the Discretized Pursuit Algorithm (DPA) by discretizing the action probability space. The DPA was shown to be superior to its continuous counterpart.…”

Section: Structural Development Of Field Of Lamentioning

confidence: 99%

See 2 more Smart Citations

The design of absorbing Bayesian pursuit algorithms and the formal analyses of their ε-optimality

Zhang

Oommen

Granmo

2016

Pattern Anal Applic

View full text Add to dashboard Cite

The fundamental phenomenon that has been used to enhance the convergence speed of Learning Automata (LA) is that of incorporating the running Maximum Likelihood (ML) estimates of the action reward probabilities into the probability updating rules for selecting the actions. The frontiers of this field have been recently expanded by replacing the ML estimates with their corresponding Bayesian counterparts that incorporate the properties of the conjugate priors [1][2][3]. These constitute the Bayesian Pursuit Algorithm (BPA) [1], and the Discretized Bayesian Pursuit Algorithm (DBPA) [2,3]. Although these algorithms have been designed and efficiently implemented 1 , and are, arguably, the fastest and most accurate LA reported in the literature 2 , the proofs of their ε-optimal convergence has been unsolved. This is precisely the intent of this paper. In this paper, we present a single unifying analysis by which the proofs of both the continuous and discretized schemes are proven. We emphasize that unlike the ML-based pursuit schemes, the Bayesian schemes have to not only consider the estimates themselves but also the distributional forms of their conjugate posteriors and their higher order moments -all of which render the proofs to be particularly challenging. As far as we know, apart from the results themselves, the methodologies of this proof have been unreported in the literature -they are both pioneering and novel. ‡ This author can be contacted at: Department of ICT, University of Agder, Grimstad, Norway. E-mail address: ole.granmo@uia.no.1 The version of the BPA presented here, namely the Absorbing Bayesian Pursuit Algorithm (ABPA), is distinct from the version presented in [1]. The reason for proposing this newer absorbing version will be explained in the body of the paper.2 The families of BPA are faster and more accurate than their counterparts that invoke the ML estimates because unlike the former which use the information in the mean, the BPA families utilize the information at a higher-end quantile (95%th precentile) of the posterior Bayesian distribution. This is the rationale for the claim that they are probably, the fastest and most accurate reported LA.

show abstract

“…In other words, the proofs of these results are already found in the literature, namely in [26] and in [23] respectively. The proofs are thus not repeated here.…”

mentioning

confidence: 54%

Section: Proofs Of Pasmentioning

confidence: 98%

See 1 more Smart Citation

The design of absorbing Bayesian pursuit algorithms and the formal analyses of their ε-optimality

Zhang

Oommen

Granmo

2016

Pattern Anal Applic

View full text Add to dashboard Cite

show abstract

“…The initial motivation for discretizing LA was to increase their rate of convergence and to avoid the need for generating real numbers with arbitrary precision [17]. In continuous algorithms, since the action probability is updated by multiplying a constant 1 − λ, the probability of selecting the optimal action can only be approached asymptotically to unity, but never be actually attained.…”

Section: Discretized Bayesian Pursuit Algorithmmentioning

confidence: 99%

Discretized Bayesian Pursuit – A New Scheme for Reinforcement Learning

Zhang

Granmo

Oommen

2012

Advanced Research in Applied Artificial Intelligence

View full text Add to dashboard Cite

Abstract. The success of Learning Automata (LA)-based estimator algorithms over the classical, Linear Reward-Inaction (L RI )-like schemes, can be explained by their ability to pursue the actions with the highest reward probability estimates. Without access to reward probability estimates, it makes sense for schemes like the L RI to first make large exploring steps, and then to gradually turn exploration into exploitation by making progressively smaller learning steps. However, this behavior becomes counter-intuitive when pursuing actions based on their estimated reward probabilities. Learning should then ideally proceed in progressively larger steps, as the reward probability estimates turn more accurate. This paper introduces a new estimator algorithm, the Discretized Bayesian Pursuit Algorithm (DBPA), that achieves this. The DBPA is implemented by linearly discretizing the action probability space of the Bayesian Pursuit Algorithm (BPA) [1]. The key innovation is that the linear discrete updating rules mitigate the counter-intuitive behavior of the corresponding linear continuous updating rules, by augmenting them with the reward probability estimates. Extensive experimental results show the superiority of DBPA over previous estimator algorithms. Indeed, the DBPA is probably the fastest reported LA to date.

show abstract

“…In practice, the relatively slow rate of convergence of these algorithms constituted a limiting factor in their applicability. In order to increase their speed of convergence, the concept of discretizing the probability space was introduced in [10][11][12]. This concept is implemented by restricting the probability of choosing an action to be one of a finite number of values in the interval [0,1].…”

Section: Introductionmentioning

confidence: 99%

A formal proof of the 𝜖-optimality of discretized pursuit algorithms

et al. 2015

View full text Add to dashboard Cite

Learning Automata (LA) can be reckoned to be the founding algorithms on which the field of Reinforcement Learning has been built. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and of these, the family of discretized algorithms are proven to converge even faster than their continuous counterparts. However, it has recently been reported that the previous proofs for ε-optimality for all the reported algorithms for the past three decades have been flawed 1 . We applaud the researchers who discovered this flaw, and who further proceeded to rectify the proof for the Continuous Pursuit Algorithm (CPA). The latter proof examines the monotonicity property of the probability of selecting the optimal action, and requires the learning parameter to be continuously changing. In this paper, we provide a new method to prove the ε-optimality of the Discretized Pursuit Algorithm (DPA) which does not require this constraint, by virtue of the fact that the DPA has, in and of itself, absorbing barriers to which the LA can jump in a discretized manner. Unlike the proof given [3] for an absorbing version of the CPA, which utilizes the single-action Hoeffding's inequality, the current proof invokes, what we shall refer to, as the "multi-action" version of the Hoeffding's inequality. We believe that our proof is both unique and pioneering. It can also form the basis for formally showing the ε-optimality of the other EAs that possess absorbing states.

show abstract

Discretized pursuit learning automata

Cited by 147 publications

References 16 publications

The design of absorbing Bayesian pursuit algorithms and the formal analyses of their ε-optimality

The design of absorbing Bayesian pursuit algorithms and the formal analyses of their ε-optimality

Discretized Bayesian Pursuit – A New Scheme for Reinforcement Learning

A formal proof of the 𝜖-optimality of discretized pursuit algorithms

Contact Info

Product

Resources

About