Generalized Resilience and Robust Statistics

Abstract

Robust statistics traditionally focuses on outliers, or perturbations in total variation distance. However, a dataset could be corrupted in many other ways, such as systematic measurement errors and missing covariates. We generalize the robust statistics approach to consider perturbations under any Wasserstein distance, and show that robust estimation is possible whenever a distribution's population statistics are robust under a certain family of friendly perturbations. This generalizes a property called resilience previously employed in the special case of mean estimation with outliers. We justify the generalized resilience property by showing that it holds under moment or hypercontractive conditions. Even in the total variation case, these subsume conditions in the literature for mean estimation, regression, and covariance estimation; the resulting analysis simplifies and sometimes improves these known results in both population limit and finite-sample rate. Our robust estimators are based on minimum distance (MD) functionals (Donoho and Liu, 1988), which project onto a set of distributions under a discrepancy related to the perturbation. We present two approaches for designing MD estimators with good finite-sample rates: weakening the discrepancy and expanding the set of distributions. We also present connections to Gao et al. (2019)'s recent analysis of generative adversarial networks for robust estimation.

Differential privacy and robust statistics in high dimensions

“…In comparison, the existing robust and DP estimator from [LKKO21], which runs in polynomial time, requires the knowledge of the covariance matrix Σ and a larger sample complexity of n = Ω((d/α 2 ) + (d 3/2 log(1/δ))/(αε)). If privacy is not required (i.e., ε = ∞), a robust mean estimator from [ZJS19] achieves the same error bound and sample complexity as ours.…”

“…If only robust error bound under α-corruption is concerned, [ZJS19] also achieves the same optimal error bound, but does not provide differential privacy. Further, in this robust but not private case with ε = ∞, our sample complexity improves by a factor of α 2/k upon the state-of-the-art sample complexity of [ZJS19,Theorem 3.3]…”

“…Definition 3.2 (Resilience for mean estimation [SCV18,ZJS19]). For some α ∈ (0, 1), ρ 1 ∈ R + , and ρ 2 ∈ R + , we say a set of n data points S good is (α, ρ 1 , ρ 2 )-resilient with respect to (µ, Σ) if for any T ⊂ S good of size |T | ≥ (1 − α)n, the following holds for all v ∈ R d with v = 1:…”

“…ρ 1 = O(α 1−1/k ) and ρ 2 = O(α 1−2/k ) (Lemma 3.15). Resilience plays a crucial role in robust statistics, where the resilience of a dataset determines the minimax sample complexity of estimating population statistics from adversarially corrupted samples [SCV18,ZJS19].…”

“…This measures how resilient the empirical mean is to subsampling or deletion of a fraction of the samples. This resilience is a central concept in robust statistical estimation when a fraction of the dataset is arbitrarily corrupted by an adversary [SCV18,ZJS19]. We show and exploit the fact that resilience is fundamentally related to the sensitivity of robust statistics.…”