Bandit Processes and Dynamic Allocation Indices

Abstract

The paper aims to give a unified account of the central concepts in recent work on bandit processes and dynamic allocation indices; to show how these reduce some previously intractable problems to the problem of calculating such indices; and to describe how these calculations may be carried out. Applications to stochastic scheduling, sequential clinical trials and a class of search problems are discussed.

Risk aversion in expected intertemporal discounted utilities bandit problems

“…Our presentation of bandit problems is quite sketchy, and we send the reader to specialized references such as (Gittins 1979(Gittins , 1989Whittle 1982;Berry and Fristedt 1985).…”

“…This durable good problem differs from the non durable good problem (discussed previously), since in the former case information acquisition has a constant cost, while in the latter it has an endogenous cost given that examination means consumption. The solution of this search model, with repetitive consumption has been worked out in clinical trials and in economics; it is known as the armed bandit problem (see Gittins 1979Gittins , 1989Berry and Fristedt 1985).…”

Optimal channel access for TCP performance improvement in cognitive radio networks

“…A relaxed version of the restless bandit problem is introduced, and the problem can be solved optimally in polynomial time. Based on this relaxed version, a priority-index heuristic policy is proposed to reduce to the optimal Gittins index policy [30] in the special case of the multi-armed bandit problem. However, the Whittle's index heuristic can be only applied when the restless bandit problem satisfies a certain indexability property, which is hard to check [21,31].…”

Stochastic online scheduling

“…He introduces a priority policy that relies on an index which can be computed for each job based on the properties of a job, but not on other jobs. Gittins [12] showed that this priority index is a special case of his Gittins index [12,13]. Twenty years after Sevcik presented the priority policy, Weiss [33] formulated Sevcik's priority index again in terms of the Gittins index and provided a different proof of the optimality of the priority policy.…”