Smoothness-Adaptive Contextual Bandits

Abstract

We study a non-parametric multi-armed bandit problem with stochastic covariates, where a key driver of complexity is the smoothness with which the payoff functions vary with covariates. Previous studies have derived minimax-optimal algorithms in cases where it is a priori known how smooth the payoff functions are. In practice, however, advance information about the smoothness of payoff functions is typically not available, and misspecification of smoothness may severely deteriorate the performance of existing methods. In this work, we consider a framework where the smoothness is not known a priori, and study when and how algorithms may adapt to unknown smoothness. First, we establish that, in general, designing bandit algorithms that adapt to the unknown smoothness of payoff functions is impossible. We overcome this impossibility result by leveraging the notion of self-similarity, a concept from the statistics literature that is traditionally invoked to enable adaptive confidence intervals. Under a self-similarity assumption, we develop a policy for inferring the smoothness of the payoff functions using observations that are collected throughout the decision-making process, and establish that this policy matches (up to a logarithmic scale) the regret rate that is achievable when smoothness is known a priori. Finally, we extend our method to account for local notions of smoothness and show that, under reasonable assumptions, our method achieves performance characterized by the local complexity of the problem as opposed to its global complexity.

Regret Minimization in Isotonic, Heavy-Tailed Contextual Bandits via Adaptive Confidence Bands

“…Is is possible to design policies that achieve the optimal regret, without explicit knowledge of the underlying smoothness?. This question was explored in [28], who discovered that this is not possible in general. They established that the absence of adaptive policies arises from the non-existence of adaptive confidence sets in non-parametric regression under smoothness classes.…”

Tractable contextual bandits beyond realizability

“…A third set of bandit algorithms that does not fall neatly into any of the two categories above are algorithms that allow for a non-parametric model class. For example, in Rigollet and Zeevi (2010), Perchet et al (2013) the reward model is assumed to be Hölder continuous but non-differentiable, and in Hu et al (2020), Gur et al (2019) it satisfies a Hölder smoothness assumption. The main characteristic of this class of algorithms is that they partition the covariate space into hypercubes of appropriate size and run multi-armed bandit algorithms within each cube.…”