Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs

Xue, Bo; Wang, Guanghui; Wang, Yimu; Zhang, Lijun

doi:10.48550/arxiv.2004.13465

Cited by 2 publications

(9 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other variations with heavy-tailed losses are also studied in the literature, e.g., linear bandits (Medina & Yang, 2016;Xue et al, 2020), contextual bandits (Shao et al, 2018) and Lipschitz bandits (Lu et al, 2019). However, none of above algorithms removes the dependency on α.…”

Section: Algorithmmentioning

confidence: 99%

Adaptive Best-of-Both-Worlds Algorithm for Heavy-Tailed Multi-Armed Bandits

Huang¹,

Yan²,

Huang³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we generalize the concept of heavy-tailed multi-armed bandits to adversarial environments, and develop robust best-of-bothworlds algorithms for heavy-tailed multi-armed bandits (MAB), where losses have α-th (1 < α ≤ 2) moments bounded by σ α , while the variances may not exist. Specifically, we design an algorithm HTINF, when the heavy-tail parameters α and σ are known to the agent, HTINF simultaneously achieves the optimal regret for both stochastic and adversarial environments, without knowing the actual environment type a-priori. When α, σ are unknown, HTINF achieves a log T -style instance-dependent regret in stochastic cases and o(T ) no-regret guarantee in adversarial cases. We further develop an algorithm AdaTINF, achieving O(σK 1− 1 /α T 1 /α ) minimax optimal regret even in adversarial settings, without prior knowledge on α and σ. This result matches the known regret lower-bound (Bubeck et al., 2013), which assumed a stochastic environment and α and σ are both known. To our knowledge, the proposed HTINF algorithm is the first to enjoy a best-of-both-worlds regret guarantee, and AdaTINF is the first algorithm that can adapt to both α and σ to achieve optimal gap-indepedent regret bound in classical heavytailed stochastic MAB setting and our novel adversarial formulation.

show abstract

Section: Algorithmmentioning

confidence: 99%

Adaptive Best-of-Both-Worlds Algorithm for Heavy-Tailed Multi-Armed Bandits

Huang¹,

Yan²,

Huang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The previous robust mean estimators (Bubeck et al, 2013), such as truncated empirical mean and median of means (Bubeck et al, 2013;Medina and Yang, 2016;Shao et al, 2018;Xue et al, 2020) which require the estimation of the mean, cannot effectively handle this super heavy-tailed noise. On the other hand, since the mean does not exist, we are also required to measure the performance of the agent with high-probability pseudo-regret (defined in Section 2.2).…”

Section: +mentioning

confidence: 99%

“…A line of recent work (Bubeck et al, 2013;Medina and Yang, 2016;Shao et al, 2018;Xue et al, 2020) on the heavy-tailed MAB or linear bandits uses the truncated empirical mean and median of means as the robust estimators. However, without assuming the finite moments of order 1 + for some ∈ (0, 1], it remains unclear whether one can attain equivalent regret/sample complexity for the more heavy-tailed setting, e.g, the noise of the payoff follows a Student's t-distribution, or whether sublinear regret algorithms of any form are even possible at all.…”

Section: Related Workmentioning

confidence: 99%

“…In comparison with existing literature, we require no assumptions on the bound or sub-Gaussianity of η, e.g., in (Abbasi-Yadkori et al, 2011). Meanwhile, we also impose no restrictions on the existence of the (1 + )-th moment of the noise, which is required in Bubeck et al (2013); Medina and Yang (2016); Shao et al (2018); Xue et al (2020). Indeed, the α -th moment of η for any α < α does exist.…”

Section: Super Heavy-tailed Noisementioning

confidence: 99%

“…Specifically, for the rewards with a finite second order moment, by utilizing more robust statistical estimators, Bubeck et al (2013) achieves regret bound of the same order as in the bounded/sub-Gaussian loss setting. Then, Medina and Yang (2016); Shao et al (2018); Xue et al (2020) study the heavy-tailed linear bandits. They consider a general characterization of heavy-tailed payoffs in bandits, where the reward takes the form r = µ + η, where µ is an unknown but fixed number and η is a random noise, whose distribution has a finite moment of order 1 + .…”

mentioning

confidence: 99%

See 2 more Smart Citations

Breaking the Moments Condition Barrier: No-Regret Algorithm for Bandits with Super Heavy-Tailed Payoffs

Zhong¹,

Huang²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite a large amount of effort in dealing with heavy-tailed error in machine learning, little is known when moments of the error can become non-existential: the random noise η satisfies Pr[|η| > |y|] ≤ 1/|y| α for some α > 0. We make the first attempt to actively handle such super heavy-tailed noise in bandit learning problems: We propose a novel robust statistical estimator, mean of medians, which estimates a random variable by computing the empirical mean of a sequence of empirical medians. We then present a generic reductionist algorithmic framework for solving bandit learning problems (including multi-armed and linear bandit problem): the mean of medians estimator can be applied to nearly any bandit learning algorithm as a black-box filtering for its reward signals and obtain similar regret bound as if the reward is sub-Gaussian. We show that the regret bound is nearoptimal even with very heavy-tailed noise. We also empirically demonstrate the effectiveness of the proposed algorithm, which further corroborates our theoretical results.

show abstract

Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs

Cited by 2 publications

References 11 publications

Adaptive Best-of-Both-Worlds Algorithm for Heavy-Tailed Multi-Armed Bandits

Adaptive Best-of-Both-Worlds Algorithm for Heavy-Tailed Multi-Armed Bandits

Breaking the Moments Condition Barrier: No-Regret Algorithm for Bandits with Super Heavy-Tailed Payoffs

Contact Info

Product

Resources

About