Optimization for Data Analysis

J., Wright, Stephen; Recht, Benjamin

doi:10.1017/9781009004282

Cited by 21 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternatively, we may view these subproblems from the point of view of machine learning, where (essentially) the same class of optimization problems is solved by (possibly) different techniques. We refer the interested reader to [54,60] for a survey of optimization methods for machine learning and data analysis problems. These techniques might be applicable very successfully at least in certain situations.…”

Section: Using the Decomposition Lmentioning

confidence: 99%

Constrained composite optimization and augmented Lagrangian methods

et al. 2023

View full text Add to dashboard Cite

We investigate finite-dimensional constrained structured optimization problems, featuring composite objective functions and set-membership constraints. Offering an expressive yet simple language, this problem class provides a modeling framework for a variety of applications. We study stationarity and regularity concepts, and propose a flexible augmented Lagrangian scheme. We provide a theoretical characterization of the algorithm and its asymptotic properties, deriving convergence results for fully nonconvex problems. It is demonstrated how the inner subproblems can be solved by off-the-shelf proximal methods, notwithstanding the possibility to adopt any solvers, insofar as they return approximate stationary points. Finally, we describe our matrix-free implementation of the proposed algorithm and test it numerically. Illustrative examples show the versatility of constrained composite programs as a modeling tool and expose difficulties arising in this vast problem class.

show abstract

Section: Using the Decomposition Lmentioning

confidence: 99%

Constrained composite optimization and augmented Lagrangian methods

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Remark 2. Note that studying convergence of lower level optimization problem, which is extensively studied in existing literature (Nesterov et al, 2018;Wright and Recht, 2022), is not the focus of this article. Assumption 5 holds for convex functions satisfying quadratic growth condition (Karimi et al, 2016) Meanwhile, Assumption (6) holds strongly convex functions, for example.…”

Section: Convergence To Stationary Pointsmentioning

confidence: 99%

Convergent First-Order Methods for Bi-level Optimization and Stackelberg Games

Maheshwari¹,

Sasty²,

Ratliff³

et al. 2023

Preprint

View full text Add to dashboard Cite

We propose an algorithm to solve a class of bi-level optimization problems using only first-order information. In particular, we focus on a class where the inner minimization has unique solutions. Unlike contemporary algorithms, our algorithm does not require the use of an oracle estimator for the gradient of the bi-level objective or an approximate solver for the inner problem. Instead, we alternate between descending on the inner problem using naïve optimization methods and descending on the upper-level objective function using specially constructed gradient estimators. We provide non-asymptotic convergence rates to stationary points of the bi-level objective in the absence of convexity of the closed-loop function and further show asymptotic convergence to only local minima of the bi-level problem. The approach is inspired by ideas from the literature on two-timescale stochastic approximation algorithms.

show abstract

“…Compared with the original problem (4), the problem in ( 5) is an unconstrained optimization problem with some simple structure so that we can apply some efficient optimization algorithms, such as stochastic gradient descent (Wright and Recht, 2022).…”

Section: A Computationally Efficient Formulationmentioning

confidence: 99%

Augmentation Invariant Manifold Learning

Wang¹

2022

Preprint

View full text Add to dashboard Cite

Data augmentation is a widely used technique and an essential ingredient in the recent advance in self-supervised representation learning. By preserving the similarity between augmented data, the resulting data representation can improve various downstream analyses and achieve state-of-art performance in many applications. To demystify the role of data augmentation, we develop a statistical framework on a lowdimension product manifold to theoretically understand why the unlabeled augmented data can lead to useful data representation. Under this framework, we propose a new representation learning method called augmentation invariant manifold learning and develop the corresponding loss function, which can work with a deep neural network to learn data representations. Compared with existing methods, the new data representation simultaneously exploits the manifold's geometric structure and invariant property of augmented data. Our theoretical investigation precisely characterizes how the data representation learned from augmented data can improve the k-nearest neighbor classifier in the downstream analysis, showing that a more complex data augmentation leads to more improvement in downstream analysis. Finally, numerical experiments on simulated and real datasets are presented to support the theoretical results in this paper.

show abstract

Optimization for Data Analysis

Cited by 21 publications

References 0 publications

Constrained composite optimization and augmented Lagrangian methods

Constrained composite optimization and augmented Lagrangian methods

Convergent First-Order Methods for Bi-level Optimization and Stackelberg Games

Augmentation Invariant Manifold Learning

Contact Info

Product

Resources

About