The Adversarial Robustness of Sampling

Ben‐Eliezer, Omri; Yogev, Eylon

doi:10.1145/3375395.3387643

Cited by 29 publications

(40 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The same argument, along with the shaping technique of Howard et al [HRMS18] yields a Bernstein-type law of iterated logarithms that controls |W t − Wt /p| at a level Õ(1/p + W t /p log log t), which should be useful more broadly. This full version (presented in §A) further shows that the 'Bernoulli-sampler' [BY20; Alo+21] offers a continuous approximation in the sense of Ben-Eliezer & Yogev [BY20], but with the error for sets of low incidence flattened as expected due to Bernstein's inequality. For our purposes, the point of Lemma 1 is to allow us to argue that no matter what the adversary does, if we uniformly abstain at a rate p, then we will 'catch' any mistake-prone function before it makes O(1/p) mistakes.…”

Section: The Adversarial Casementioning

confidence: 92%

“…The above analysis was inspired by studying the recent work of Ben-Eliezer and Yogev [BY20], on adversarial sketching -their goal was to maintain an estimate of the incidence of a process within a given set (and more generally, within sets in a given system) while using limited memory, and they analysed a similar sampling approach, showing via an application of Freedman's inequality that [BY20, Lemma 4.1]…”

Section: A2 An Improved Alln Via a Self-normalised Law Of Iterated Lo...mentioning

confidence: 99%

“…This essentially amounts to using the crude bound W T ≤ T . The same paper, in Theorem 1.4 and associated lemmata argues that the Reservoir Sampler [BY20, §2] of size ∼ pT controls deviations uniformly over time at scale T p log log T δ , and it was asserted that the Bernoulli Sampler cannot attain such a 'continuous robustness' [BY20,§1]. The above result improves upon this in a few ways -firstly, the result applies to the simpler Bernoulli sampler, and improves the deviation control to O( √ W t ) instead of O( √ T ).…”

Section: A2 An Improved Alln Via a Self-normalised Law Of Iterated Lo...mentioning

confidence: 99%

“…For the adversarial case, the analysis of the scheme relies on a new 'adversarial uniform law of large numbers'(ALLN) to argue that such methods cannot incur too many mistakes. This ALLN uses a self-normalised martingale concentration bound, and further yields an adaptive continuous approximation guarantee for the Bernoulli-sampling sketch in the sense of Ben-Eliezer & Yogev [BY20;Alo+21]. The theoretical exploration is complemented by illustrative experiments that implement our scheme on two benchmark datasets.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Online Selective Classification with Limited Feedback

Gangrade¹,

Kag²,

Cutkosky³

et al. 2021

Preprint

View full text Add to dashboard Cite

Motivated by applications to resource-limited and safety-critical domains, we study selective classification in the online learning model, wherein a predictor may abstain from classifying an instance. For example, this may model an adaptive decision to invoke more resources on this instance. Two salient aspects of the setting we consider are that the data may be non-realisable, due to which abstention may be a valid long-term action, and that feedback is only received when the learner abstains, which models the fact that reliable labels are only available when the resource intensive processing is invoked.Within this framework, we explore strategies that make few mistakes, while not abstaining too many times more than the best-in-hindsight error-free classifier from a given class. That is, the one that makes no mistakes, while abstaining the fewest number of times. We construct simple versioning-based schemes for any µ ∈ (0, 1], that make most T µ mistakes while incurring Õ(T 1−µ ) excess abstention against adaptive adversaries. We further show that this dependence on T is tight, and provide illustrative experiments on realistic datasets.

show abstract

Section: The Adversarial Casementioning

confidence: 92%

Section: A2 An Improved Alln Via a Self-normalised Law Of Iterated Lo...mentioning

confidence: 99%

Section: A2 An Improved Alln Via a Self-normalised Law Of Iterated Lo...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Online Selective Classification with Limited Feedback

Gangrade¹,

Kag²,

Cutkosky³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The size of the sample needed is the VC dimension of the set system of the sub-populations of interest. Ben-Eliezer and Yogev [3] showed that when sampling from a stream, if the sample is public and an adversary may choose the stream based on the samples so far, then the VC Dimension may not be enough. Alon, Ben-Eliezer, Dagan, Moran, Naor and Yogev [1] showed that the Littlestone dimension, which might be much larger than the VC dimension, captures the size of the sample needed in this case.…”

Section: Related Workmentioning

confidence: 99%

Keep That Card in Mind: Card Guessing with Limited Memory

Menuhin¹,

Naor²

2021

Preprint

View full text Add to dashboard Cite

A card guessing game is played between two players, Guesser and Dealer. At the beginning of the game, the Dealer holds a deck of n cards (labeled 1, ..., n). For n turns, the Dealer draws a card from the deck, the Guesser guesses which card was drawn, and then the card is discarded from the deck. The Guesser receives a point for each correctly guessed card.With perfect memory, a Guesser can keep track of all cards that were played so far and pick at random a card that has not appeared so far, yielding in expectation ln n correct guesses. With no memory, the best a Guesser can do will result in a single guess in expectation.We consider the case of a memory bounded Guesser that has m < n memory bits. We show that the performance of such a memory bounded Guesser depends much on the behavior of the Dealer. In more detail, we show that there is a gap between the static case, where the Dealer draws cards from a properly shuffled deck or a prearranged one, and the adaptive case, where the Dealer draws cards thoughtfully, in an adversarial manner. Specifically:1. We show a Guesser with O(log 2 n) memory bits that scores a near optimal result against any static Dealer.2. We show that no Guesser with m bits of memory can score better than O( √ m) correct guesses, thus, no Guesser can score better than min{ √ m, ln n}, i.e., the above Guesser is optimal.3. We show an efficient adaptive Dealer against which no Guesser with m memory bits can make more than ln m + 2 ln log n + O(1) correct guesses in expectation.These results are (almost) tight, and we prove them using compression arguments that harness the guessing strategy for encoding.

show abstract