On the Global Convergence of Particle Swarm Optimization Methods

Abstract

In this paper we provide a rigorous convergence analysis for the renowned Particle Swarm Optimization method using tools from stochastic calculus and the analysis of partial differential equations. Based on a time-continuous formulation of the particle dynamics as a system of stochastic differential equations, we establish the convergence to a global minimizer in two steps. First, we prove the consensus formation of the dynamics by analyzing the time-evolution of the variance of the particle distribution. Consecutively, we show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. Our results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum are satisfied. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

Consensus based optimization with memory effects: random selection and applications

“…Owing to the convergence analysis of CBO algorithms [3,9,10,19] and recent analysis of PSO [14,20] we are able to prove convergence of the algorithm under mild assumption on the objective function. This is done by first approximating the algorithm with a continuous-intime dynamics and secondly by giving a probabilistic description to the particles system.…”

“…Such model is then analyzed to recover convergence guarantees under precise assumption on the objective function. Developed in the field of statistical physics, this approach has shown be fruitful in studying particle-based metaheuristic algorithms [9,10,20]. We note that convergence in mean-field law was recently proved in [37] in an independent work.…”

An adaptive consensus based method for multi-objective optimization with uniform Pareto front approximation

“…From a mathematical viewpoint, this class of metaheuristic methods is inspired by the corresponding mean-field dynamics based on particle swarming and multi-agent social interactions, which have been widely used to study complex systems in life sciences, social sciences and economics [17,43,44,46,53]. These techniques have proven fruitful to demonstrate convergence towards a global minimum for single-objective problems, not only in the case of CBO methods, but also for the popular Particle Swarm Optimization (PSO) algorithm [33,38], thus paving the way to provide a mathematical foundation for other metaheuristics.…”