Sequential Experimental Design for Transductive Linear Bandits

Abstract

In this paper we introduce the transductive linear bandit problem: given a set of measurement vectors X ⊂ R d , a set of items Z ⊂ R d , a fixed confidence δ, and an unknown vector θ * ∈ R d , the goal is to infer argmax z∈Z z θ * with probability 1 − δ by making as few sequentially chosen noisy measurements of the form x θ * as possible. When X = Z, this setting generalizes linear bandits, and when X is the standard basis vectors and Z ⊂ {0, 1} d , combinatorial bandits. Such a transductive setting naturally arises when the set of measurement vectors is limited due to factors such as availability or cost. As an example, in drug discovery the compounds and dosages X a practitioner may be willing to evaluate in the lab in vitro due to cost or safety reasons may differ vastly from those compounds and dosages Z that can be safely administered to patients in vivo. Alternatively, in recommender systems for books, the set of books X a user is queried about may be restricted to well known best-sellers even though the goal might be to recommend more esoteric titles Z. In this paper, we provide instance-dependent lower bounds for the transductive setting, an algorithm that matches these up to logarithmic factors, and an evaluation. In particular, we provide the first non-asymptotic algorithm for linear bandits that nearly achieves the information theoretic lower bound.

Selective Sampling for Online Best-arm Identification

“…Note that this is precisely the number of labels required in the pool-based setting where the agent can choose any x ∈ X that she desires at each time t (e.g. [11]). In the other extreme, E[U] = ρ(ν) log(1/δ) so that the constraint in the label complexity E[L] is equivalent to ρ(ν) ≥ λ/ν ∞ ρ(λ).…”

“…The proof of Theorem 1 relies on standard techniques from best arm identification lower bounds (see e.g. [17,11]).…”

“…Best-Arm Identification and Online Experimental Design. Our techniques are based on experimental design methods for best-arm identification in linear bandits, see [24,11,5]. In the setting of these works, there exists a pool of examples X and at each time any x ∈ X can be selected with replacement.…”

“…. , x τ ∼ λ and y i = x i , θ * + i where i is iid 1 sub-Gaussian noise, then there exists an estimator θ := θ({(x i , y i )} τ i=1 ) such that θ, z * > max z∈Z\z * θ, z with probability at least 1 − δ [11]. In particular, τ ≥ ρ(λ) log(|Z|/δ) samples suffice to guarantee that arg max z∈Z θ, z = arg max z∈Z θ * , z =: z * .…”

“…By Lemma 2, we know that P(E) ≥ 1 − δ. Then, the rest of the proof is the same as the one in [11], but we include it here for completeness. Assume that E holds.…”