No abstract

When developing and analyzing new hyperparameter optimization (HPO) methods, it is vital to empirically evaluate and compare them on well-curated benchmark suites. In this work, we list desirable properties and requirements for such benchmarks and propose a new set of challenging and relevant multifidelity HPO benchmark problems motivated by these requirements. For this, we revisit the concept of surrogate-based benchmarks and empirically compare them to more widely-used tabular benchmarks, showing that the latter ones may induce bias in performance estimation and ranking of HPO methods. We present a new surrogate-based benchmark suite for multifidelity HPO methods consisting of 9 benchmark collections that constitute over 700 multifidelity HPO problems in total. All our benchmarks also allow for querying of multiple optimization targets, enabling the benchmarking of multi-objective HPO. We examine and compare our benchmark suite with respect to the defined requirements and show that our benchmarks provide viable additions to existing suites. * Equal contribution Preprint. Under review.

IntroductionDouble inversion recovery (DIR) has been validated as a sensitive magnetic resonance imaging (MRI) contrast in multiple sclerosis (MS). Deep learning techniques can use basic input data to generate synthetic DIR (synthDIR) images that are on par with their acquired counterparts. As assessment of longitudinal MRI data is paramount in MS diagnostics, our study's purpose is to evaluate the utility of synthDIR longitudinal subtraction imaging for detection of disease progression in a multicenter data set of MS patients.MethodsWe implemented a previously established generative adversarial network to synthesize DIR from input T1-weighted and fluid-attenuated inversion recovery (FLAIR) sequences for 214 MRI data sets from 74 patients and 5 different centers. One hundred and forty longitudinal subtraction maps of consecutive scans (follow-up scan—preceding scan) were generated for both acquired FLAIR and synthDIR. Two readers, blinded to the image origin, independently quantified newly formed lesions on the FLAIR and synthDIR subtraction maps, grouped into specific locations as outlined in the McDonald criteria.ResultsBoth readers detected significantly more newly formed MS-specific lesions in the longitudinal subtractions of synthDIR compared with acquired FLAIR (R1: 3.27 ± 0.60 vs 2.50 ± 0.69 [P = 0.0016]; R2: 3.31 ± 0.81 vs 2.53 ± 0.72 [P < 0.0001]). Relative gains in detectability were most pronounced in juxtacortical lesions (36% relative gain in lesion counts—pooled for both readers). In 5% of the scans, synthDIR subtraction maps helped to identify a disease progression missed on FLAIR subtraction maps.ConclusionsGenerative adversarial networks can generate high-contrast DIR images that may improve the longitudinal follow-up assessment in MS patients compared with standard sequences. By detecting more newly formed MS lesions and increasing the rates of detected disease activity, our methodology promises to improve clinical decision-making.

Despite all the benefits of automated hyperparameter optimization (HPO), most modern HPO algorithms are black-boxes themselves. This makes it difficult to understand the decision process which lead to the selected configuration, reduces trust in HPO, and thus hinders its broad adoption. Here, we study the combination of HPO with interpretable machine learning (IML) methods such as partial dependence plots. However, if such methods are naively applied to the experimental data of the HPO process in a post-hoc manner, the underlying sampling bias of the optimizer can distort interpretations. We propose a modified HPO method which efficiently balances the search for the global optimum w.r.t. predictive performance and the reliable estimation of IML explanations of an underlying black-box function by coupling Bayesian optimization and Bayesian Algorithm Execution. On benchmark cases of both synthetic objectives and HPO of a neural network, we demonstrate that our method returns more reliable explanations of the underlying black-box without a loss of optimization performance.Preprint. Under review.

Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks.However, the process of designing HPO algorithms is still an unsystematic and manual process: New algorithms are often built on top of prior work, where limitations are identified and improvements are proposed. Even though this approach is guided by expert knowledge, it is still somewhat arbitrary. The process rarely allows for gaining a holistic understanding of which algorithmic components drive performance and carries the risk of overlooking good algorithmic design choices.We present a principled approach to automated benchmark-driven algorithm design applied to multi-fidelity HPO (MF-HPO). First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to, common existing HPO algorithms and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration (in some ways, simpler than established methods) performs very well as long as some critical configuration parameters are set to the right value.