Design of Experiments for Stochastic Contextual Linear Bandits

Almost Optimal Batch-Regret Tradeoff for Batch Linear Contextual Bandits

“…Note that if the context set is deterministic, this objective corresponds to (the square root of) the G-optimality criterion in classical experimental designs (see, e.g., [Pukelsheim, 2006, Atkinson et al, 2007). For stochastic context sets, the objective was recently found closely related to linear contextual bandits and studied by [Ruan et al, 2020, Zanette et al, 2021. Our ExplorationPolicy procedure is employed during each batch of the main algorithm to decide the policy used in the next batch.…”

“…In contrast, thanks to a few key technical modifications and a new matrix concentration inequality (to be introduced in Section 1.1.4), our algorithm can directly learn the desired exploration policy with better performance, and is simpler to describe and implement. We also note that in the concurrent work [Zanette et al, 2021], the authors studied a similar task to our ExplorationPolicy. In Section 4.1, we compare our performance guarantee and the results in [Zanette et al, 2021], and demonstrate the superiority of our procedure.…”

“…We also note that in the concurrent work [Zanette et al, 2021], the authors studied a similar task to our ExplorationPolicy. In Section 4.1, we compare our performance guarantee and the results in [Zanette et al, 2021], and demonstrate the superiority of our procedure.…”

“…Existing matrix concentration inequalities (see, e.g., [Tropp, 2012]) play an important role in recent works on batch linear contextual bandits [Ruan et al, 2020, Zanette et al, 2021. For example, the proof techniques of Theorem 5.1 in [Tropp, 2012] may yield the following concentration bound in Proposition 1 (and the upper bound on k X k may be similarly derived).…”

“…To the best of our knowledge, the state-of-the-art regret upper bound for the adaptive-context setting is O(d T ln(KT )) [Abbasi-Yadkori et al, 2011]. In this work (as well as the most related works [Han et al, 2020, Ruan et al, 2020, Zanette et al, 2021 on batch linear contextual bandits), we focus on a particularly useful case in the non-adaptive-context setting, namely the stochastic contexts. In this case, the context sets at each time step are independently generated from a pre-defined (but unknown) distribution D. In many real-world applications such as clinical trial and recommendation system, the patients or customers can often be viewed as independent samples from the population and therefore stochastic contexts are a natural abstraction of these practical scenarios.…”