Consensus-based optimisation with truncated noise

Abstract

Consensus-based optimisation (CBO) is a versatile multi-particle metaheuristic optimisation method suitable for performing non-convex and non-smooth global optimisations in high dimensions. It has proven effective in various applications while at the same time being amenable to a theoretical convergence analysis. In this paper, we explore a variant of CBO, which incorporates truncated noise in order to enhance the well-behavedness of the statistics of the law of the dynamics. By introducing this additional truncation in the noise term of the CBO dynamics, we achieve that, in contrast to the original version, higher moments of the law of the particle system can be effectively bounded. As a result, our proposed variant exhibits enhanced convergence performance, allowing in particular for wider flexibility in choosing the noise parameter of the method as we confirm experimentally. By analysing the time evolution of the Wasserstein- $2$ distance between the empirical measure of the interacting particle system and the global minimiser of the objective function, we rigorously prove convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialisation. Numerical evidences demonstrate the benefit of truncating the noise in CBO.

CBX: Python and Julia Packages for Consensus-Based Interacting Particle Methods

“…However, unlike many such methods, CBX has been designed to be compatible with rigorous convergence analysis at the mean-field level (the infinite-particle limit, see Huang & Qiu, 2022). Many convergence results have been shown, whether in the original formulation (Carrillo et al, 2018;Fornasier, Klock, et al, 2021), for CBO with anisotropic noise (Carrillo et al, 2021;Fornasier et al, 2022), with memory effects (Riedl, 2023), with truncated noise (Fornasier et al, 2024), for polarised CBO (Bungert et al, 2024), and PSO (Huang et al, 2023). The relation between CBO and stochastic gradient descent has been recently established by Riedl et al (2023), which suggests a previously unknown yet fundamental connection between derivative-free and gradient-based approaches.…”