Randomized Sketches of Convex Programs With Sharp Guarantees

Abstract

Random projection (RP) is a classical technique for reducing storage and computational costs. We analyze RP-based approximations of convex programs, in which the original optimization problem is approximated by the solution of a lower-dimensional problem. Such dimensionality reduction is essential in computation-limited settings, since the complexity of general convex programming can be quite high (e.g., cubic for quadratic programs, and substantially higher for semidefinite programs). In addition to computational savings, random projection is also useful for reducing memory usage, and has useful properties for privacy-sensitive optimization. We prove that the approximation ratio of this procedure can be bounded in terms of the geometry of constraint set. For a broad class of random projections, including those based on various sub-Gaussian distributions as well as randomized Hadamard and Fourier transforms, the data matrix defining the cost function can be projected down to the statistical dimension of the tangent cone of the constraints at the original solution, which is often substantially smaller than the original dimension. We illustrate consequences of our theory for various cases, including unconstrained and ℓ 1 -constrained least squares, support vector machines, low-rank matrix estimation, and discuss implications on privacy-sensitive optimization and some connections with denoising and compressed sensing.

Weighted Gradient Coding with Leverage Score Sampling

“…Other methods for approximating the leverage scores are available [17][18][19], that do not require directly computing the singular vectors. We also note that a multitude of other efficient constructions of sketching matrices and iterative sketching algorithms have been studied in the literature [20][21][22][23].…”

Non-Gaussian Random Matrices on Sets:Optimal Tail Dependence and Applications

“…In essence, a randomized sketch reduces the dimension of the original optimization problem through random projections, which can be beneficial in both computational time and memory storage. Following the problem formulation and ideas in [25], consider convex program in the form of…”

“…We will use an argument similar to [25] to prove Theorem 5.2. First let us state a deterministic result that says δ-optimality can be obtained by controlling two quantities.…”

“…First let us state a deterministic result that says δ-optimality can be obtained by controlling two quantities. Lemma 5.3 (Lemma 1 [25]). For any sketching matrix A ∈ R m×n , let…”

“…However, in many applications, non-gaussian random mappings are more useful because of their computational/storage benefits or simply the difficulty to generate Gaussian matrices using sampling devices [17]. For example, sparse or structured random matrices are preferred in both dimension reduction [8] and random sketching in numerical linear algebra [1,14,25,34] since they provide more efficient matrix multiplications than dense and unstructured matrices such as Gaussian ones. Certain formulations in compressed sensing also naturally require random matrices such as randomly subsampled Fourier measurements [18] or Bernoulli random matrices [28].…”