Improving Accuracy of Interpretability Measures in Hyperparameter Optimization via Bayesian Algorithm Execution

Abstract

Despite all the benefits of automated hyperparameter optimization (HPO), most modern HPO algorithms are black-boxes themselves. This makes it difficult to understand the decision process which lead to the selected configuration, reduces trust in HPO, and thus hinders its broad adoption. Here, we study the combination of HPO with interpretable machine learning (IML) methods such as partial dependence plots. However, if such methods are naively applied to the experimental data of the HPO process in a post-hoc manner, the underlying sampling bias of the optimizer can distort interpretations. We propose a modified HPO method which efficiently balances the search for the global optimum w.r.t. predictive performance and the reliable estimation of IML explanations of an underlying black-box function by coupling Bayesian optimization and Bayesian Algorithm Execution. On benchmark cases of both synthetic objectives and HPO of a neural network, we demonstrate that our method returns more reliable explanations of the underlying black-box without a loss of optimization performance.Preprint. Under review.

Multipoint-BAX: a new approach for efficiently tuning particle accelerator emittance via virtual objectives

“…In order to reduce the number of function queries, our MULTIPOINT-BAX method uses techniques from an information-based BAX method, InfoBAX [32,58,59], to make targeted queries that maximize the mutual-information between O A and the next observation y t . The InfoBAX procedure is a sequential algorithm that seeks to maximize the acquisition function, defined here as the expected information gain (EIG) about O A upon observing y t .…”

Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning

“…The field of human-centered HPO/AutoML has gained increasing traction in the last years with approaches targeted at explaining the hyperparameter optimization process to increase the trust in automated tools (Pfisterer, Thomas, and Bischl 2019;Moosbauer et al 2021Moosbauer et al , 2022Segel et al 2023). This also includes concrete tools developed to help a user interpret the results such as XAutoML (Zöller et al 2022) or DeepCave (Sass et al 2022).…”

“…With this paper, we propose an interactive humancentered HPO (Pfisterer, Thomas, and Bischl 2019;Souza et al 2021;Moosbauer et al 2021;Hvarfner et al 2022;Moosbauer et al 2022;Francia, Giovanelli, and Pisano 2023;Segel et al 2023;Mallik et al 2023) approach for MO-ML algorithms that frees users from choosing a predefined quality indicator suitable for their needs by learning one tailored towards them based on feedback. To achieve this, it first learns the desired Pareto front shape from the user in a short interactive session and then starts a corresponding HPO process optimizing towards the previously…”

Combining Bayesian optimization and automation to simultaneously optimize reaction conditions and routes

“…One key challenge of incorporating BO methods in R&D is that these methods are typically considered as black boxes with limited explainability and interpretability, 35 hindering their widespread adoption. Additionally, when the search space is large, researchers face difficulties in visualizing and understanding the way that the parameters influence the objectives.…”