Statistical Distances and Their Role in Robustness

Abstract

Statistical distances, divergences, and similar quantities have a large history and play a fundamental role in statistics, machine learning and associated scientific disciplines. However, within the statistical literature, this extensive role has too often been played out behind the scenes, with other aspects of the statistical problems being viewed as more central, more interesting, or more important. The behind the scenes role of statistical distances shows up in estimation, where we often use estimators based on minimizing a distance, explicitly or implicitly, but rarely studying how the properties of a distance determine the properties of the estimators. Distances are also prominent in goodness-of-fit, but the usual question we ask is "how powerful is this method against a set of interesting alternatives" not "what aspect of the distance between the hypothetical model and the alternative are we measuring?"Our focus is on describing the statistical properties of some of the distance measures we have found to be most important and most visible. We illustrate the robust nature of Neyman's chi-squared and the non-robust nature of Pearson's chi-squared statistics and discuss the concept of discretization robustness.

Statistical Distances and the Construction of Evidence Functions for Model Adequacy

“…where a(t) is a suitable probability mass function (see Lindsay, 1994;Markatou et al, 2017). For example, when a(t) = m(t) and τ (t) = d(t) the proportion of observations in the sample with value t, we obtain Pearson's chi-squared distance.…”

“…Other choices of a(t) result in different members of the chi-squared family. The family of chi-squared distances has a very clear interpretation as a risk measure (Lindsay, 2004;Markatou et al, 2017). First, the chi-squared distance is obtained as the solution of an optimization problem with interpretable constraints.…”

“…In general, the chisquared distances are affected by outliers. However, a member of this class, the symmetric chi-squared distance is obtained if we use in place of a(t) the mixture 0.5m(t) + 0.5d(t), and provides estimators that are unaffected by outliers (see Markatou et al, 1998;Markatou et al, 2017;Markatou and Chen, 2018). An attractive characteristic of the symmetric chi-squared distance is that it admits a testing interpretation.…”

“…An attractive characteristic of the symmetric chi-squared distance is that it admits a testing interpretation. For details, see Markatou et al (2017). The fact that it is possible to obtain the chi-squared distances as solutions of a certain optimization problem with interpretable as a variance constraint, allows us by analogy to the construction of Scheffé's confidence intervals for parameter contrasts, to interpret chi-squared distances as tools that permit the construction of "Scheffé-type" confidence intervals for models.…”

Pruning Algorithms for Low-Dimensional Non-metric k-NN Search: A Case Study

“…Statistical distances in general and, KL divergence in particular, play an important role in ML [8,17]. They are typically non-symmetric.…”