A derivative-free method for structured optimization problems

Abstract

Structured optimization problems are ubiquitous in fields like data science and engineering. The goal in structured optimization is using a prescribed set of points, called atoms, to build up a solution that minimizes or maximizes a given function. In the present paper, we want to minimize a black-box function over the convex hull of a given set of atoms, a problem that can be used to model a number of real-world applications. We focus on problems whose solutions are sparse, i.e., solutions that can be obtained as a proper convex combination of just a few atoms in the set, and propose a suitable derivative-free inner approximation approach that nicely exploits the structure of the given problem. This enables us to properly handle the dimensionality issues usually connected with derivative-free algorithms, thus getting a method that scales well in terms of both the dimension of the problem and the number of atoms. We analyze global convergence to stationary points. Moreover, we show that, under suitable assumptions, the proposed algorithm identifies a specific subset of atoms with zero weight in the final solution after finitely many iterations. Finally, we report numerical results showing the effectiveness of the proposed method.

Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization

“…Large-Scale DFO There have been several alternative approaches considered for improving the scalability of DFO. These often consider problems with specific structure which enable efficient model construction, such as partial separability [24,60], sparse Hessians [4], and minimization over the convex hull of finitely many points [27]. On the other hand, there is a growing body of literature on 'gradient sampling' techniques for machine learning problems.…”

Scalable Derivative-Free Optimization for Nonlinear Least-Squares Problems

“…This type of problem arises across a broad range of application areas (Conn et al, 2009;Audet & Hare, 2017), but has attracted particular recent attention in the learning community for problems such as black-box attacks (Chen et al, 2017;Ughi et al, 2019), hyperparameter tuning (Ghanbari & Scheinberg, 2017;Lakhmiri et al, 2020) and reinforcement learning (Mania et al, 2018;Choromanski et al, 2019). A cur-rent deficiency of DFO methods is their performance on large-scale problems, which is critical to their utility in machine learning; there have been several recent works aimed at improving the scalability of DFO (Bergou et al, 2019;Roberts, 2019;Porcelli & Toint, 2020;Cristofari & Rinaldi, 2020).…”