A formal proof of the 𝜖-optimality of discretized pursuit algorithms

Abstract

Learning Automata (LA) can be reckoned to be the founding algorithms on which the field of Reinforcement Learning has been built. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and of these, the family of discretized algorithms are proven to converge even faster than their continuous counterparts. However, it has recently been reported that the previous proofs for ε-optimality for all the reported algorithms for the past three decades have been flawed 1 . We applaud the researchers who discovered this flaw, and who further proceeded to rectify the proof for the Continuous Pursuit Algorithm (CPA). The latter proof examines the monotonicity property of the probability of selecting the optimal action, and requires the learning parameter to be continuously changing. In this paper, we provide a new method to prove the ε-optimality of the Discretized Pursuit Algorithm (DPA) which does not require this constraint, by virtue of the fact that the DPA has, in and of itself, absorbing barriers to which the LA can jump in a discretized manner. Unlike the proof given [3] for an absorbing version of the CPA, which utilizes the single-action Hoeffding's inequality, the current proof invokes, what we shall refer to, as the "multi-action" version of the Hoeffding's inequality. We believe that our proof is both unique and pioneering. It can also form the basis for formally showing the ε-optimality of the other EAs that possess absorbing states.

A new methodology for identifying unreliable sensors in data fusion

“…The proof of Conjecture 1 is beyond the aim of this article and we allude to the proofs reported in [39] as a potential possible way to justify it.…”

“…Inspired by the family of pursuit LA algorithm [23,2,39], we design a novel pursuit LA for S-Model that pursues the action that has the highest "average reward" among the two actions. Please note that the classical pursuit LA found in the literature [23,2,39] operate only with binary feedback while our scheme uses continuous feedback. Now, we shall provide the details of our Pursuit S-LA algorithm that is discretized.…”

A Conclusive Analysis of the Finite-Time Behavior of the Discretized Pursuit Learning Automaton

“…Though the work in [24] was innovative and compass both the CPA and the DPA, there was a flaw in its reasoning, which rendered the analysis incorrect. To correct this flaw, new proofs for the CPA and DPA's convergence were proposed in [36] and [35] respectively, where the authors investigated the submartingale property of the probability of selecting the optimal action, and invoked the theory of regular functions to prove that the PAs converge to the optimal action in probability. They also claimed that the new proof methodology could be extended to prove the convergence of other Pursuit-based LAs.…”

A pursuit learning solution to underwater communications with limited mobility agents

“…Our algorithm is inspired by the family of pursuit LA algorithm [1,15,26]. However, instead of pursuing the action with the highest reward among the offered actions, we pursue the action that leads to an increase in the reward compared to the previously visited state at time instant t − 1…”