Bandits with switching costs

Abstract

We study the adversarial multi-armed bandit problem in a setting where the player incurs a unit cost each time he switches actions. We prove that the player's T -round minimax regret in this setting is Θ(T 2/3 ), thereby closing a fundamental gap in our understanding of learning with bandit feedback. In the corresponding full-information version of the problem, the minimax regret is known to grow at a much slower rate of Θ( √ T ). The difference between these two rates provides the first indication that learning with bandit feedback can be significantly harder than learning with fullinformation feedback (previous results only showed a different dependence on the number of actions, but not on T .)In addition to characterizing the inherent difficulty of the multi-armed bandit problem with switching costs, our results also resolve several other open problems in online learning. One direct implication is that learning with bandit feedback against bounded-memory adaptive adversaries has a minimax regret of Θ(T 2/3 ). Another implication is that the minimax regret of online learning in adversarial Markov decision processes (MDPs) is Θ(T 2/3 ). The key to all of our results is a new randomized construction of a multi-scale random walk, which is of independent interest and likely to prove useful in additional settings.

SpecWatch: A framework for adversarial spectrum monitoring with unknown statistics

“…3) We prove that the expected weak regret of SpecWatch-II is O(T 2/3 ), which matches the lower bound in [25]. Therefore, SpecWatch-II is asymptotically optimal.…”

Online Learning with Graph-Structured Feedback against Adaptive Adversaries

“…. in a similar fashion as in Dekel et al [6], upon which we build our adaptive loss sequences with memory of size 1. Formally, Algorithm 5 in Appendix D.1 is used to generate the oblivious loss sequences L 1:T .…”

“…Unfortunately, although the Exp3.G algorithm is minimax optimal for all strongly-observable graphs when the adversary is oblivious, this is not true when the game involves switching costs. It is known that certain strategies exist which incur O( √ T ) regret in the full-information game, whereas the multi-armed bandit problem suffers a lower bound of Ω(T 2/3 ) [6], even though both games are induced by strongly-observable graphs. Hence, some strongly-observable games are more difficult than others.…”

“…By the enumeration in Figure 3 of Appendix D, such a graph must contain a two-node subgraph G 1 that preserves the observability of G. By Lemma 10 in Appendix D.3, the game induced by G is at least as hard as any game on G 1 . The game on G 1 is simply a bandit problem, so it has Ω(T 2/3 ) regret [6]. The Ω(T 2/3 ) regret lower bound for weakly-observable games against oblivious opponents naturally extends to games against adaptive opponents.…”

“…A specific class of unit-memory adversaries of particular interest corresponds to oblivious adversaries with switching costs. Although the minimax regret was shown to be Θ(T 1/2 ) in the case of full-information games and Θ(T 2/3 ) in the case of bandit feedback [5,6], the gap between O(T 2/3 ) upper bounds and Ω(T 1/2 ) lower bounds for the more general class of adversaries with unit memory in the case of full-information feedback has remained unaddressed. For the problem of general feedback graphs with oblivious adversaries, Alon et al [7,8] showed that the regret is characterized by certain characteristics of the graph structure involving domination numbers and independent sets.…”