Lean From Thy Neighbor: Stochastic & Adversarial Bandits in a Network

Abstract

An individual's decisions are often guided by those of his or her peers, i.e., neighbors in a social network. Presumably, being privy to the experiences of others aids in learning and decision making, but how much advantage does an individual gain by observing her neighbors? Such problems make appearances in sociology and economics and, in this paper, we present a novel model to capture such decision-making processes and appeal to the classical multi-armed bandit framework to analyze it. Each individual, in addition to her own actions, can observe the actions and rewards obtained by her neighbors, and can use all of this information in order to minimize her own regret. We provide algorithms for this setting, both for stochastic and adversarial bandits, and show that their regret smoothly interpolates between the regret in the classical bandit setting and that of the full-information setting as a function of the neighbors' exploration. In the stochastic setting the additional information must simply be incorporated into the usual estimation of the rewards, while in the adversarial setting this is attained by constructing a new unbiased estimator for the rewards and appropriately bounding the amount of additional information provided by the neighbors. These algorithms are optimal up to log factors; despite the fact that the agents act independently and selfishly, this implies that it is an approximate Nash equilibria for all agents to use our algorithms. Further, we show via empirical simulations that our algorithms, often significantly, outperform existing algorithms that one could apply to this setting.

Quantifying the Burden of Exploration and the Unfairness of Free Riding

“…Works by Bastani, Bayati, and Khosravi [2]; Kannan, Morgenstern, Roth, Waggoner, and Wu [9]; and Raghavan, Slivkins, Vaughan, and Wu [14] show that if the data is sufficiently diverse, e.g., if the contexts are randomly perturbed, then exploration may not be necessary. Celis and Salehi [7] consider a model in both the stochastic and the adversarial setting where each agent in the network plays a certain zero-regret algorithm (UCB in the stochastic setting and EXP3 in the adversarial setting) and study how much information an agent can gather from his neighbors.…”