Fast Bayesian optimization of Needle-in-a-Haystack problems using zooming memory-based initialization (ZoMBI)

Abstract

Needle-in-a-Haystack problems exist across a wide range of applications including rare disease prediction, ecological resource management, fraud detection, and material property optimization. A Needle-in-a-Haystack problem arises when there is an extreme imbalance of optimum conditions relative to the size of the dataset. However, current state-of-the-art optimization algorithms are not designed with the capabilities to find solutions to these challenging multidimensional Needle-in-a-Haystack problems, resulting in slow convergence or pigeonholing into a local minimum. In this paper, we present a Zooming Memory-Based Initialization algorithm, entitled ZoMBI, that builds on conventional Bayesian optimization principles to quickly and efficiently optimize Needle-in-a-Haystack problems in both less time and fewer experiments. The ZoMBI algorithm demonstrates compute time speed-ups of 400× compared to traditional Bayesian optimization as well as efficiently discovering optima in under 100 experiments that are up to 3× more highly optimized than those discovered by similar methods.

In Pursuit of the Exceptional: Research Directions for Machine Learning in Chemical and Materials Science

“…167 Zooming-based Bayesian optimizations have a similar alternation between global sampling and local optimization. 41 behavior, in which the search process is characterized as a random walk with a heavy-tailed distribution of step sizes, and which in practice looks like local exploration in a region interspersed by large jumps to new regions. 168 V.D.…”

“…Empirically, many successful materials ML problems are approximately smooth and convex response surfaces, with a broad basin of attraction toward a few local optima, like the schematic example plotted in Figure a. Thus, it is unsurprising that ML-based approaches for representing the landscape can be successful, and iterative optimization is an efficient strategy.…”

“…(a) Previous work has focused on optimizations on smooth, convex response surfaces; (b) Exceptional material properties are often characterized by very sharp discontinuities as composition and reaction conditions are changed. Adapted from ref .…”

“…Another framework is provided by the Multidimensional Archive of Phenotypic Elites (MAP-Elites) algorithm, an evolutionary algorithm used in reinforcement learning, which samples and stores multiple candidate solutions (“elites”) on a grid to preserve a diverse set of characteristics for possible solution . Zooming-based Bayesian optimizations have a similar alternation between global sampling and local optimization . This is also reminiscent of Lévy flight models of animal foraging behavior, in which the search process is characterized as a random walk with a heavy-tailed distribution of step sizes, and which in practice looks like local exploration in a region interspersed by large jumps to new regions …”

PAL 2.0: a physics-driven bayesian optimization framework for material discovery

“…[13][14][15][16] It has been used to optimize a wide range of problems, including automatic algorithm configuration, automatic machine learning toolboxes, and optimization of combinatorial spaces for materials and drug discovery. 10,[17][18][19][20][21][22][23][24] A BayesOpt algorithm essentially requires two sets of functions: (i) a ''surrogate model'' for the objective function and (ii) an ''acquisition function'' that is updated, based on the surrogate model, to provide a recommendation for the next candidate to explore. Some typical examples of surrogate models include Gaussian Processes, 25 random forests 26 and Bayesian Neural Networks.…”