Statistical Analysis of Wasserstein Distributionally Robust Estimators

Abstract

We consider statistical methods which invoke a min-max distributionally robust formulation to extract good out-of-sample performance in data-driven optimization and learning problems. Acknowledging the distributional uncertainty in learning from limited samples, the min-max formulations introduce an adversarial inner player to explore unseen covariate data. The resulting Distributionally Robust Optimization (DRO) formulations, which include Wasserstein DRO formulations (our main focus), are specified using optimal transportation phenomena. Upon describing how these infinite-dimensional min-max problems can be approached via a finite-dimensional dual reformulation, the tutorial moves into its main component, namely, explaining a generic recipe for optimally selecting the size of the adversary's budget. This is achieved by studying the limit behavior of an optimal transport projection formulation arising from an inquiry on the smallest confidence region that includes the unknown population risk minimizer. Incidentally, this systematic prescription coincides with those in specific examples in high-dimensional statistics and results in error bounds that are free from the curse of dimensions. Equipped with this prescription, we present a central limit theorem for the DRO estimator and provide a recipe for constructing compatible confidence regions that are useful for uncertainty quantification. The rest of the tutorial is devoted to insights into the nature of the optimizers selected by the min-max formulations and additional applications of optimal transport projections.

The Geometry of Adversarial Training in Binary Classification

“…In contrast to problem (1.1), the effect of explicit regularization on the robustness of models is less direct, but this is compensated by a richer structure that can be used to study the theoretical properties of their solutions more directly. The connections between adversarial training and regularization have been intensely explored in recent years in the context of classical parametric learning settings; see [1,9,15] and references within. For example, when θ ∈ Θ = R d represents the parameters of a linear regression model and the loss function for the model is the squared loss, the following identity holds , where G p is an optimal transport distance of the form…”

“…In particular, in this setting ⋅ q becomes the regularization term R, the risk functional is Ĵ = √ J, and λ = √ ε. Through an equivalence like (1.3) it is possible to motivate new ways of calibrating regularization parameters in models with a convex loss function (where first order optimality conditions guarantee global optimality) as has been done in [1]. Beyond linear regression, the equivalence between adversarial training and regularization problems has also been studied in parametric binary classification settings such as logistic regression and SVMs (see [1]), as well as in distributionally robust grouped variable selection, and distributionally robust multi-output learning (see [9]).…”