Calibration of Distributionally Robust Empirical Optimization Models

Abstract

In this paper, we study the out-of-sample properties of robust empirical optimization and develop a theory for data-driven calibration of the "robustness parameter" for worst-case maximization problems with concave reward functions. Building on the intuition that robust optimization reduces the sensitivity of the expected reward to errors in the model by controlling the spread of the reward distribution, we show that the first-order benefit of "little bit of robustness" is a significant reduction in the variance of the out-of-sample reward while the corresponding impact on the mean is almost an order of magnitude smaller. One implication is that a substantial reduction in the variance of the out-of-sample reward (i.e. sensitivity of the expected reward to model misspecification) is possible at little cost if the robustness parameter is properly calibrated. To this end, we introduce the notion of a robust mean-variance frontier to select the robustness parameter and show that it can be approximated using resampling methods like the bootstrap. Our examples also show that "open loop" calibration methods (e.g. selecting a 90% confidence level regardless of the data and objective function) can lead to solutions that are very conservative out-of-sample.

Robust Wasserstein profile inference and applications to machine learning

“…In particular, as we discuss in the case of LASSO, according to our results corresponding to contribution E), RWPI based prescription of the size of uncertainty actually can be shown (under suitable regularity conditions) to decay at rate O( log d/n) (uniformly over d and n such that log 2 d n), which is in agreement with the findings of high-dimensional statistics literature (see [13,30,3] and references therein). A profile function based approach towards calibrating the radius of uncertainty in the context of empirical likelihood based DRO can be found in [27,15,21,26].…”

On the Impossibility of Statistically Improving Empirical Optimization: A Second-Order Stochastic Dominance Perspective

“…Putting aside the technical differences in using L-DRO versus the common DRO in (18) (the latter requires an extra layer of analysis in the Lagrangian reformulation), we point out two conceptual distinctions between Gotoh et al (2018Gotoh et al ( , 2021 and our results in Section 2 and 3.2. First is the criterion in measuring the quality of an obtained solution x, in particular the role of the variance of the loss function h(x, ξ), V ar P (h(x, ξ)).…”

“…In particular, any risk-aware consideration should be already incorporated into the construction of the loss h. When an obtained solution x is used in many future test cases, an estimate of Z(x), using n test test data points, has a variance given by V ar P (h(x, ξ))/n test (instead of V ar P (h(x, ξ))), and thus the variance of h plays a relatively negligible role. This is different from Gotoh et al (2018Gotoh et al ( , 2021 who takes an alternate view that puts more weight on the variability of the loss function.…”

“…Consider the expected value minimization problem min x∈X {Z(x) = E P [h(x, ξ)]} where h is some loss function. Gotoh et al (2018Gotoh et al ( , 2021) studies a regularized formulation using φ-divergence as the penalty, defined as (in the notation of the current paper):…”

“…The main insight from Gotoh et al (2018Gotoh et al ( , 2021) is a desirable improvement on the bias-variance tradeoff using the solution obtained in ( 28), which we call a "Lagrangian (L)-DRO" solution xL−DRO,λ n . More precisely, Gotoh et al ( 2018) concludes (Theorem 5.1 therein) that…”