Olympus: a benchmarking framework for noisy optimization and experiment planning

Abstract

Research challenges encountered across science, engineering, and economics can frequently be formulated as optimization tasks. In chemistry and materials science, recent growth in laboratory digitization and automation has sparked interest in optimization-guided autonomous discovery and closed-loop experimentation. Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms to achieve desired experimentation goals with the minimum number of trials. However, the experiment planning strategy that is most suitable to a scientific discovery task is a priori unknown while rigorous comparisons of different strategies are highly time and resource demanding. As optimization algorithms are typically benchmarked on low-dimensional synthetic functions, it is unclear how their performance would translate to noisy, higher-dimensional experimental tasks encountered in chemistry and materials science. We introduce Olympus, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms against realistic experiments emulated via probabilistic deep-learning models. Olympus includes a collection of experimentally derived benchmark sets from chemistry and materials science and a suite of experiment planning strategies that can be easily accessed via a user-friendly Python interface. Furthermore, Olympus facilitates the integration, testing, and sharing of custom algorithms and user-defined datasets. In brief, Olympus mitigates the barriers associated with benchmarking optimization algorithms on realistic experimental scenarios, promoting data sharing and the creation of a standard framework for evaluating the performance of experiment planning strategies.

Chemputation and the Standardization of Chemical Informatics

“…Recently two new approaches to chemical optimization were reported, Summit 133 and Olympus, 134 benchmarking frameworks offer a large set of optimization strategies together with virtual benchmarks, and an experimental planning toolkit. In contrast to the aforementioned software, these frameworks are fully open-sourced and due to their open interface, it is possible to plug in any given algorithm, and this could be universally adapted to any typical automation platform.…”

Golem: an algorithm for robust experiment and process optimization

“…An optimization campaign thus typically proceeds by iteratively testing sets of parameters x, as defined via a design of experiment or as suggested by an experiment planning algorithm [25][26][27] . Common design of experiment approaches rely on random or systematic searches of parameter combinations.…”

“…Specifically, we consider the calibration of an HPLC protocol, in which six controllable parameters (Figure 7a, section S.3) can be varied to maximize the peak area, i.e., the amount of drawn sample reaching the detector. 25,65 Imagine we ran 1386 experiments in which we tested combinations of these six parameters at random. The experiment with the largest peak area provides the best set of parameters found.…”

FitBenchmarking: an open source Python package comparing data fitting software

“…Tools such as Paver (Bussieck et al, 2014), part of the COIN-OR initiative, can be used alongside CUTEst (or other tools) for this purpose. The packages Olympus (Häse et al, 2021) and Benchopt (BenchOpt 1.1.0, 2021) have been recently developed as benchmarking and analysis frameworks for optimization problems. Olympus is designed for experiment planning and provides analytic benchmark problems, experimental datasets, and emulated datasets, but could be adapted to be applied to any optimization (or data-fitting) problem.…”