Learning Automata-Based Solutions to the Nonlinear Fractional Knapsack Problem With Applications to Optimal Resource Allocation

2016

Pattern Anal Applic

Self Cite

The fundamental phenomenon that has been used to enhance the convergence speed of Learning Automata (LA) is that of incorporating the running Maximum Likelihood (ML) estimates of the action reward probabilities into the probability updating rules for selecting the actions. The frontiers of this field have been recently expanded by replacing the ML estimates with their corresponding Bayesian counterparts that incorporate the properties of the conjugate priors [1][2][3]. These constitute the Bayesian Pursuit Algorithm (BPA) [1], and the Discretized Bayesian Pursuit Algorithm (DBPA) [2,3]. Although these algorithms have been designed and efficiently implemented 1 , and are, arguably, the fastest and most accurate LA reported in the literature 2 , the proofs of their ε-optimal convergence has been unsolved. This is precisely the intent of this paper. In this paper, we present a single unifying analysis by which the proofs of both the continuous and discretized schemes are proven. We emphasize that unlike the ML-based pursuit schemes, the Bayesian schemes have to not only consider the estimates themselves but also the distributional forms of their conjugate posteriors and their higher order moments -all of which render the proofs to be particularly challenging. As far as we know, apart from the results themselves, the methodologies of this proof have been unreported in the literature -they are both pioneering and novel. ‡ This author can be contacted at: Department of ICT, University of Agder, Grimstad, Norway. E-mail address: ole.granmo@uia.no.1 The version of the BPA presented here, namely the Absorbing Bayesian Pursuit Algorithm (ABPA), is distinct from the version presented in [1]. The reason for proposing this newer absorbing version will be explained in the body of the paper.2 The families of BPA are faster and more accurate than their counterparts that invoke the ML estimates because unlike the former which use the information in the mean, the BPA families utilize the information at a higher-end quantile (95%th precentile) of the posterior Bayesian distribution. This is the rationale for the claim that they are probably, the fastest and most accurate reported LA.

Section: Applications Of Lamentioning

confidence: 99%

The design of absorbing Bayesian pursuit algorithms and the formal analyses of their ε-optimality

Zhang

2016

Pattern Anal Applic

Self Cite

“…They have been used in game playing [6,7], parameter optimization [8,9], vehicle path control [10], channel selection in cognitive radio networks [11], assigning capacities in prioritized networks [12], and resource allocation [13]. LA have also been used in natural language processing, string taxonomy [14], graph patitioning [15], and map learning [16].…”

Section: Learning Automata and Their Applicationsmentioning

confidence: 99%

Discretized Bayesian Pursuit – A New Scheme for Reinforcement Learning

Zhang

Advanced Research in Applied Artificial Intelligence

2012

Self Cite

Abstract. The success of Learning Automata (LA)-based estimator algorithms over the classical, Linear Reward-Inaction (L RI )-like schemes, can be explained by their ability to pursue the actions with the highest reward probability estimates. Without access to reward probability estimates, it makes sense for schemes like the L RI to first make large exploring steps, and then to gradually turn exploration into exploitation by making progressively smaller learning steps. However, this behavior becomes counter-intuitive when pursuing actions based on their estimated reward probabilities. Learning should then ideally proceed in progressively larger steps, as the reward probability estimates turn more accurate. This paper introduces a new estimator algorithm, the Discretized Bayesian Pursuit Algorithm (DBPA), that achieves this. The DBPA is implemented by linearly discretizing the action probability space of the Bayesian Pursuit Algorithm (BPA) [1]. The key innovation is that the linear discrete updating rules mitigate the counter-intuitive behavior of the corresponding linear continuous updating rules, by augmenting them with the reward probability estimates. Extensive experimental results show the superiority of DBPA over previous estimator algorithms. Indeed, the DBPA is probably the fastest reported LA to date.

“…The concept of discretizing the probability space was pioneered by Thathachar and Oommen in their study on Reward-Inaction LA [16], and since then that it has catalyzed a significant research in the design of discretized LA [1,5,9,3,4]. Recently, there has been an upsurge of research interest in solving resource allocation problems based on novel discretized LA [3,4]. In [3,4], the authors proposed a solution to the class of Stochastic Nonlinear Fractional Knapsack problems where resources had to be allocated based on incomplete and noisy information.…”

Section: State-of-the-artmentioning

confidence: 99%

“…Recently, there has been an upsurge of research interest in solving resource allocation problems based on novel discretized LA [3,4]. In [3,4], the authors proposed a solution to the class of Stochastic Nonlinear Fractional Knapsack problems where resources had to be allocated based on incomplete and noisy information. The latter solution was applied to resolve the web-polling problem, and to the problem of determining the optimal size required for estimation.…”

Section: State-of-the-artmentioning

confidence: 99%

A Stochastic Search on the Line-Based Solution to Discretized Estimation

Yazidi

Advanced Research in Applied Artificial Intelligence

2012

Self Cite

Abstract. Recently, Oommen and Rueda [11] presented a strategy by which the parameters of a binomial/multinomial distribution can be estimated when the underlying distribution is nonstationary. The method has been referred to as the Stochastic Learning Weak Estimator (SLWE), and is based on the principles of continuous stochastic Learning Automata (LA). In this paper, we consider a new family of stochastic discretized weak estimators pertinent to tracking time-varying binomial distributions. As opposed to the SLWE, our proposed estimator is discretized, i.e., the estimate can assume only a finite number of values. It is well known in the field of LA that discretized schemes achieve faster convergence speed than their corresponding continuous counterparts. By virtue of discretization, our estimator realizes extremely fast adjustments of the running estimates by jumps, and it is thus able to robustly, and very quickly, track changes in the parameters of the distribution after a switch has occurred in the environment. The design principle of our strategy is based on a solution, pioneered by Oommen [7], for the Stochastic Search on the Line (SSL) problem. The SSL solution proposed in [7], assumes the existence of an Oracle which informs the LA whether to go "right" or "left". In our application domain, in order to achieve efficient estimation, we have to first infer (or rather simulate) such an Oracle. In order to overcome this difficulty, we rather intelligently construct an "Artificial Oracle" that suggests whether we are to increase the current estimate or to decrease it. The paper briefly reports conclusive experimental results that demonstrate the ability of the proposed estimator to cope with non-stationary environments with a high adaptation rate, and with an accuracy that depends on its resolution. The results which we present are, to the best of our knowledge, the first reported results that resolve the problem of discretized weak estimation using a SSL-based solution.