Sample average approximation with heavier tails I: non-asymptotic bounds with weak assumptions and stochastic constraints

Abstract

We derive exponential finite-sample nonasymptotic deviation inequalities for the sample average approximation (SAA) estimator's near-optimal solution set and optimal value. In that respect, we give three contributions. First, our bounds do not require sub-Gaussian assumptions as in previous literature of stochastic optimization (SO). Instead, we just assume random Hölder continuity and a heavy-tailed distribution with finite 2nd moments, a framework more suited for risk-averse portfolio optimization. Second, we derive new deviation inequalities for SO problems with expected-valued stochastic constraints which guarantee joint near feasibility and optimality in terms of the original problem. Unlike previous works, we do not assume a metric regular (MR) solution set, strong growth conditions on the objective nor an indirect problem reformulation (such as penalization or first order conditions, both of which are often necessary but not sufficient conditions). Instead, we just assume a MR feasible set, making our analysis general enough for many classes of problems. A finite-sample near feasibility and optimality deviation is established for MR sets which are nonconvex or which are convex and non-strictly feasible. For strictly feasible convex sets, we obtain finite-sample exact feasibility and near optimality guarantees. The feasible set's MR constant is an additional condition number. For convex sets, we present localized feasibility guarantees in terms of smaller metric entropies. Third, we obtain a general uniform concentration inequality for heavy-tailed Hölder continuous functions using empirical process theory. This is the main tool in our analysis but it is also a result of independent interest.

On Monte-Carlo methods in convex stochastic optimization

“…min x∈X f (x)) by Guigues, Juditsky, and Nemirovski in [9]. The two recent papers by Oliveira and Thompson [28,29] which have been mentioned at the end of Section 2 contain results that are closest to ours.…”

“…N in detail, let us compare the outcome of Theorem 2.9 with the current state of the art. Our focus is on recent non-asymptotic results by Oliveira and Thompson [28,29]; a general literature review will be presented in Section 2.5.…”

Diametrical Risk Minimization: Theory and Computations

“…It is clear from the proof of the theorem that the assumptions about independent sampling and subgaussian random variables can be relaxed. We only needed that the error in R m (w) relative to R(w) can be bounded for a finite number of w; see [5,23] for possible extensions.…”

A Data Efficient and Feasible Level Set Method for Stochastic Convex Optimization with Expectation Constraints

“…Traditional approaches for solving SOECs can lead to large data complexity. For instance, consider the popular strategy of replacing each expectation in (1) by a sample average approximation (SAA; Shapiro 2013, Oliveira andThompson 2017) and solving the resulting model using a deterministic iterative method (see, e.g., Nesterov 2004, Soheili andPena 2012, andreferences therein). If the number of samples used to construct SAAs is small, the solution from the deterministic approximation may be highly infeasible to the original SOEC, in addition to being suboptimal (Shapiro 2013, Oliveira andThompson 2017).…”

“…For instance, consider the popular strategy of replacing each expectation in (1) by a sample average approximation (SAA; Shapiro 2013, Oliveira andThompson 2017) and solving the resulting model using a deterministic iterative method (see, e.g., Nesterov 2004, Soheili andPena 2012, andreferences therein). If the number of samples used to construct SAAs is small, the solution from the deterministic approximation may be highly infeasible to the original SOEC, in addition to being suboptimal (Shapiro 2013, Oliveira andThompson 2017). Instead, if a large number of samples are used in each SAA, then the data complexity becomes large because the gradient or objective function evaluation at each iteration requires using a significant portion of each of the data sets.…”