Solving Stochastic Nonlinear Resource Allocation Problems Using a Hierarchy of Twofold Resource Allocation Automata

Granmo, Ole‐Christoffer; Oommen, B. John

doi:10.1109/tc.2009.189

Cited by 51 publications

(69 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, the quantity x m , as defined in (16), which renders m (P ) to be superregular, makes the φ m (P ) to be subregular. Therefore, according to (11),…”

Section: U (P ) = E[ (P (T + 1))|p (T) = P ]mentioning

confidence: 99%

“…LA have found applications in a variety of fields, including game playing [3,4], parameter optimization [5], solving knapsack-like problems and utilizing the solution in web polling and sampling [6], vehicle path control [7], assigning capacities in prioritized networks [8], and stochastically optimally allocating limited resources [6,[9][10][11]. LA have also been used in natural language processing, string taxonomy [12], graph patitioning [13], map learning [14], service selection in stochastic environments [15], numerical optimization [16], web crawling [17], microassembly path planning [18], multiagent learning [19], and in batch sequencing and sizing in just-in-time manufacturing systems [20].…”

mentioning

confidence: 99%

See 1 more Smart Citation

A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions

et al. 2014

Self Cite

View full text Add to dashboard Cite

The most difficult part in the design and analysis of Learning Automata (LA) consists of the formal proofs of their convergence accuracies. The mathematical techniques used for the different families (Fixed Structure, Variable Structure, Discretized etc.) are quite distinct. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and within this family, the set of Pursuit algorithms have been considered to be the pioneering schemes. Informally, if the environment is stationary, their ε-optimality is defined as their ability to converge to the optimal action with an arbitrarily large probability, if the learning parameter is sufficiently small/large. The existing proofs of all the reported EAs follow the same fundamental principles, and to clarify this, in the interest of simplicity, we shall concentrate on the family of Pursuit algorithms. Recently, it has been reported Ryan and Omkar (J Appl Probab 49(3): [795][796][797][798][799][800][801][802][803][804][805] 2012) that the previous proofs for ε-optimality of all the reported EAs have a common flaw.The flaw lies in the condition which apparently supports the so-called "monotonicity" property of the probability of selecting the optimal action, which states that after some time instant t 0 , the reward probability estimates will be ordered correctly forever. The authors of the various proofs have rather offered a proof for the fact that the reward probability estimates are ordered correctly at a single point of time after t 0 , which, in turn, does not guarantee the ordering forever, rendering the previous proofs incorrect. While in Ryan and Omkar (J Appl Probab 49(3):795-805, 2012), a rectified proof was presented to prove the ε-optimality of the Continuous Pursuit Algorithm (CPA), which was the pioneering EA, in this paper, a new proof is provided for the Absorbing CPA (ACPA), i.e., an algorithm which follows the CPA paradigm but which artificially has absorbing states whenever any action probability is arbitrarily close to unity. Unlike the previous flawed proofs, instead of examining the monotonicity property of the action probabilities, it rather examines their submartingale property, and then, unlike the traditional approach, invokes the theory of Regular functions to prove that the probability of converging to the optimal action can be made arbitrarily close to unity. We believe that the proof is both unique and pioneering, and adds insights into the convergence of different EAs. It can also form the basis for formally demonstrating the ε-optimality of other Estimator algorithms which are artificially rendered absorbing.

show abstract

“…Therefore, the quantity x m , as defined in (16), which renders m (P ) to be superregular, makes the φ m (P ) to be subregular. Therefore, according to (11),…”

Section: U (P ) = E[ (P (T + 1))|p (T) = P ]mentioning

confidence: 99%

mentioning

confidence: 99%

A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions

et al. 2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…The concept of discretizing the probability space was pioneered by Thathachar and Oommen in their study on Reward-Inaction LA [16], and since then that it has catalyzed a significant research in the design of discretized LA [1,5,9,3,4]. Recently, there has been an upsurge of research interest in solving resource allocation problems based on novel discretized LA [3,4]. In [3,4], the authors proposed a solution to the class of Stochastic Nonlinear Fractional Knapsack problems where resources had to be allocated based on incomplete and noisy information.…”

Section: State-of-the-artmentioning

confidence: 99%

“…Recently, there has been an upsurge of research interest in solving resource allocation problems based on novel discretized LA [3,4]. In [3,4], the authors proposed a solution to the class of Stochastic Nonlinear Fractional Knapsack problems where resources had to be allocated based on incomplete and noisy information. The latter solution was applied to resolve the web-polling problem, and to the problem of determining the optimal size required for estimation.…”

Section: State-of-the-artmentioning

confidence: 99%

A Stochastic Search on the Line-Based Solution to Discretized Estimation

Yazidi

Granmo

Oommen

2012

Advanced Research in Applied Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

Abstract. Recently, Oommen and Rueda [11] presented a strategy by which the parameters of a binomial/multinomial distribution can be estimated when the underlying distribution is nonstationary. The method has been referred to as the Stochastic Learning Weak Estimator (SLWE), and is based on the principles of continuous stochastic Learning Automata (LA). In this paper, we consider a new family of stochastic discretized weak estimators pertinent to tracking time-varying binomial distributions. As opposed to the SLWE, our proposed estimator is discretized, i.e., the estimate can assume only a finite number of values. It is well known in the field of LA that discretized schemes achieve faster convergence speed than their corresponding continuous counterparts. By virtue of discretization, our estimator realizes extremely fast adjustments of the running estimates by jumps, and it is thus able to robustly, and very quickly, track changes in the parameters of the distribution after a switch has occurred in the environment. The design principle of our strategy is based on a solution, pioneered by Oommen [7], for the Stochastic Search on the Line (SSL) problem. The SSL solution proposed in [7], assumes the existence of an Oracle which informs the LA whether to go "right" or "left". In our application domain, in order to achieve efficient estimation, we have to first infer (or rather simulate) such an Oracle. In order to overcome this difficulty, we rather intelligently construct an "Artificial Oracle" that suggests whether we are to increase the current estimate or to decrease it. The paper briefly reports conclusive experimental results that demonstrate the ability of the proposed estimator to cope with non-stationary environments with a high adaptation rate, and with an accuracy that depends on its resolution. The results which we present are, to the best of our knowledge, the first reported results that resolve the problem of discretized weak estimation using a SSL-based solution.

show abstract

“…LA have also been used in vehicle path control [18], assigning capacities in prioritized networks [19], resource allocation [20], string taxonomy [21], graph partitioning [22], and map learning [23]. To exemplify the importance of the DPA, it is worth mentioning an application where one of its variants has been applied.…”

Section: Introductionmentioning

confidence: 99%

A formal proof of the 𝜖-optimality of discretized pursuit algorithms

et al. 2015

Self Cite

View full text Add to dashboard Cite

Learning Automata (LA) can be reckoned to be the founding algorithms on which the field of Reinforcement Learning has been built. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and of these, the family of discretized algorithms are proven to converge even faster than their continuous counterparts. However, it has recently been reported that the previous proofs for ε-optimality for all the reported algorithms for the past three decades have been flawed 1 . We applaud the researchers who discovered this flaw, and who further proceeded to rectify the proof for the Continuous Pursuit Algorithm (CPA). The latter proof examines the monotonicity property of the probability of selecting the optimal action, and requires the learning parameter to be continuously changing. In this paper, we provide a new method to prove the ε-optimality of the Discretized Pursuit Algorithm (DPA) which does not require this constraint, by virtue of the fact that the DPA has, in and of itself, absorbing barriers to which the LA can jump in a discretized manner. Unlike the proof given [3] for an absorbing version of the CPA, which utilizes the single-action Hoeffding's inequality, the current proof invokes, what we shall refer to, as the "multi-action" version of the Hoeffding's inequality. We believe that our proof is both unique and pioneering. It can also form the basis for formally showing the ε-optimality of the other EAs that possess absorbing states.

show abstract

Solving Stochastic Nonlinear Resource Allocation Problems Using a Hierarchy of Twofold Resource Allocation Automata

Cited by 51 publications

References 23 publications

A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions

A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions

A Stochastic Search on the Line-Based Solution to Discretized Estimation

A formal proof of the 𝜖-optimality of discretized pursuit algorithms

Contact Info

Product

Resources

About