A Stochastic Consensus Method for Nonconvex Optimization on the Stiefel Manifold

Kim, Jeongho; Kang, Myeongju; Kim, Dohyun; Ha, Seung‐Yeal; Yang, Insoon

doi:10.1109/cdc42340.2020.9304325

Cited by 17 publications

(13 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CBO algorithms implement the update rule (2) (or possibly variations thereof, depending on the problem) and compute v α ( ρ N T ) as a guess for the global minimizer. This concept of optimization has been explored and analyzed in several recent works [9,10,14,15,19,25,33], even in a high-dimensional or non-Euclidean setting. As an example for their applicability on high-dimensional problems, we refer for instance to [9], where the authors illustrate the use of CBO for training a shallow neural network classifier for MNIST, or to [15], where (2) and ( 4) are adapted to the sphere S d−1 and achieve near state-of-the-art performance on phase retrieval and robust subspace detection problems.…”

Section: Introductionmentioning

confidence: 99%

Consensus-based optimization methods converge globally

Fornasier¹,

Klock²,

Riedl³

2021

Preprint

View full text Add to dashboard Cite

In this paper we study consensus-based optimization (CBO), which is a metaheuristic derivative-free optimization method that can globally minimize nonconvex nonsmooth functions and is amenable to theoretical analysis. Based on an experimentally supported intuition that CBO performs a gradient descent on the convex envelope of a given objective, we derive a novel technique for proving the convergence to the global minimizer in mean-field law for a rich class of objective functions. Our results unveil internal mechanisms of CBO that are responsible for the success of the method. Furthermore, we improve prior analyses by requiring minimal assumptions about the initialization of the method and by covering objectives that are merely locally Lipschitz continuous. As a by-product of our analysis, we establish a quantitative nonasymptotic Laplace principle, which may be of independent interest.

show abstract

Section: Introductionmentioning

confidence: 99%

Consensus-based optimization methods converge globally

Fornasier¹,

Klock²,

Riedl³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…which means that X α (µ N t ) is a global best location at time t. It has been proved that CBO can guarantee global convergence under suitable assumptions [16] and it is a powerful and robust method to solve many interesting non-convex high-dimensional optimization problems in machine learning [7,14]. By now, CBO methods have also been generalized to optimization over manifolds [13][14][15]22] and several variants have been explored, which use additionally, for instance, personal best information [30], binary interaction dynamics [3] or connect CBO with Particle Swarm Optimization [8,19]. The readers are referred to [31] for a comprehensive review on the recent developments of the CBO methods.…”

Section: Introductionmentioning

confidence: 99%

On the mean-field limit for the consensus-based optimization

Huang,

Qiu

2021

Preprint

View full text Add to dashboard Cite

This paper is concerned with the large particle limit for the consensus-based optimization (CBO), which was postulated in the pioneering works [6,28]. In order to solve this open problem, we adapt a compactness argument by first proving the tightness of the empirical measures {µ N } N≥2 associated to the particle system and then verifying that the limit measure µ is the unique weak solution to the mean-field CBO equation. Such results are extended to the model of particle swarm optimization (PSO).

show abstract

“…It has been proved that CBO is a powerful and robust method to solve many interesting non-convex highdimensional optimization problems in machine learning [13]. By now, CBO methods have also been generalized to optimization over manifolds [20][21][22]35]. The objective of the present paper is to complete a theory gap suggested in [25] by providing a rigorous proof of the zero-inertia limit.…”

Section: Introductionmentioning

confidence: 98%

Zero-Inertia Limit: from Particle Swarm Optimization to Consensus Based Optimization

Cipriani¹,

Huang²,

Qiu³

2021

Preprint

View full text Add to dashboard Cite

Recently a continuous description of the particle swarm optimization (PSO) based on a system of stochastic differential equations was proposed by Grassi and Pareschi in [25] where the authors formally showed the link between PSO and the consensus based optimization (CBO) through zero-inertia limit. This paper is devoted to solving this theoretical open problem proposed in [25] by providing a rigorous derivation of CBO from PSO through the limit of zero inertia, and a quantified convergence rate is obtained as well. The proofs are based on a probabilistic approach by investigating the weak convergence of the corresponding stochastic differential equations (SDEs) of Mckean type in the continuous path space and the results are illustrated with some numerical examples.

show abstract

A Stochastic Consensus Method for Nonconvex Optimization on the Stiefel Manifold

Cited by 17 publications

References 26 publications

Consensus-based optimization methods converge globally

Consensus-based optimization methods converge globally

On the mean-field limit for the consensus-based optimization

Zero-Inertia Limit: from Particle Swarm Optimization to Consensus Based Optimization

Contact Info

Product

Resources

About