Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Bischl, Bernd; Binder, Martin; Lang, Michel; Pielok, Tobias; Richter, Jakob; Coors, Stefan; Thomas, Janek; Ullmann, Theresa; Becker, Marc; Boulesteix, Anne‐Laure; Deng, Difan; Lindauer, Marius

doi:10.1002/widm.1484

Cited by 222 publications

(98 citation statements)

References 114 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The hyperparameter spaces for mlr3 were initially selected following Bischl et al (19) but then modi ed for this study. Due to time constraints and limited computational power, the number of hyperparameter con gurations tested for each training run was set to 5.…”

Section: Model De Nitionmentioning

confidence: 99%

A comparative patient-level prediction study in OMOP CDM demonstrates advantages for native and web-based implementations

Ahmadi

Nguyễn

Sedlmayr

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: The emergence of collaborations, which standardize and combine multiple clinical databases across different regions, provides a wealthy source of data, which is fundamental for clinical prediction models, such as patient-level predictions. With the aid of such large data pools, researchers are able to develop clinical prediction models for improved disease classification, risk assessment, and beyond. To fully utilize this potential, Machine Learning (ML) methods are commonly required to process these large amounts of data on disease-specific patient cohorts. As a consequence, the Observational Health Data Sciences and Informatics (OHDSI) collaborative develops a framework to facilitate the application of ML models for these standardized patient datasets by using the Observational Medical Outcomes Partnership (OMOP) common data model (CDM). Motivation: In this study, we compare the feasibility of current web-based OHDSI approaches, namely ATLAS and “Patient-level Prediction” (PLP), against a native solution (R based) to conduct such ML-based patient-level prediction analyses in OMOP. This will enable potential users to select the most suitable approach for their investigation. Methods: Each of the applied ML solutions was individually utilized to solve the same patient-level prediction task. Both approaches went through an exemplary benchmarking analysis to assess the weaknesses and strengths of the PLP R-Package. In this work, the performance of this package was subsequently compared versus the commonly used native R-package called Machine Learning in R 3 (mlr3), and its sub-packages. The approaches were evaluated on performance, execution time, availability of tools for handling imbalanced datasets, and ease of model implementation. Results: The results show that the PLP package has shorter execution times, which indicates great scalability, as well as intuitive code implementation, and numerous possibilities for visualization. However, limitations in comparison to native packages were depicted in the implementation of specific ML classifiers (e.g., Lasso), which may result in a decreased performance for real-world prediction problems. Conclusion: The ﬁndings here contribute to the overall effort of developing ML-based prediction models on a clinical scale and provide a snapshot for future studies that explicitly aim to develop patient-level prediction models in OMOP CDM.

show abstract

Section: Model De Nitionmentioning

confidence: 99%

A comparative patient-level prediction study in OMOP CDM demonstrates advantages for native and web-based implementations

Ahmadi

Nguyễn

Sedlmayr

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Deep neural networks lie at the heart of many of the artificial intelligence applications that are ubiquitous in our society. Over the past several years, methods for training these networks have become more automatic [1][2][3][4][5], but still remain more an art than a science. This paper introduces the high-level concept of general cyclical training as another step in making it easier to optimally train neural networks.…”

Section: Introductionmentioning

confidence: 99%

General Cyclical Training of Neural Networks

Smith¹

2023

AAIML

View full text Add to dashboard Cite

This position paper describes the principle of “General Cyclical Training” in machine learning, where training starts and ends with “easy training” and the “hard training” happens during the middle epochs. We propose several manifestations for training neural networks, including algorithmic examples (via hyper-parameters and loss functions), data-based examples, and model-based examples. Specifically, we introduce several novel techniques: cyclical weight decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature, cyclical data augmentation, cyclical gradient clipping, and cyclical semi-supervised learning. In addition, we demonstrate that cyclical weight decay, cyclical softmax temperature, and cyclical gradient clipping (as three examples of this principle) are beneficial in the test accuracy performance of a trained model. Furthermore, we discuss model-based examples (such as pretraining and knowledge distillation) from the perspective of general cyclical training and recommend some changes to the typical training methodology. In summary, this paper defines the general cyclical training concept and discusses several specific ways in which this concept can be applied to training neural networks. In the spirit of reproducibility, the code used in our experiments is available at https://github.com/lnsmith54/CFL.

show abstract

“…On the other hand, a variety of significant progresses concerning NAS and hyper-parameters optimisation (HPO) for deep neural networks have been made in recent years [11][12][13][14][15][16][17][18]. For example, Xie et al [14] employed a GA to optimise the network structure encoded in a fixed-length binary string, and the experimental results on image classification datasets, including MNIST, CIFAR10 and ILSVRC2012, are promising.…”

Section: Introductionmentioning

confidence: 99%

“…On the other hand, a variety of significant progresses concerning NAS and hyper‐parameters optimisation (HPO) for deep neural networks have been made in recent years [11–18]. For example, Xie et al.…”

Section: Introductionmentioning

confidence: 99%

SOPA‐GA‐CNN: Synchronous optimisation of parameters and architectures by genetic algorithms with convolutional neural network blocks for securing Industrial Internet‐of‐Things

Huang

Zeng

Geng

et al. 2023

IET Cyber-Syst and Robotics

View full text Add to dashboard Cite

In recent years, deep learning has been applied to a variety of scenarios in Industrial Internet of Things (IIoT), including enhancing the security of IIoT. However, the existing deep learning methods utilised in IIoT security are manually designed by heavily relying on the experience of the designers. The authors have made the first contribution concerning the joint optimisation of neural architecture search and hyper-parameters optimisation for securing IIoT. A novel automated deep learning method called synchronous optimisation of parameters and architectures by GA with CNN blocks (SOPA-GA-CNN) is proposed to synchronously optimise the hyperparameters and block-based architectures in convolutional neural networks (CNNs) by genetic algorithms (GA) for the intrusion detection issue of IIoT. An efficient hybrid encoding strategy and the corresponding GA-based evolutionary operations are designed to characterise and evolve both the hyperparameters, including batch size, learning rate, weight optimiser and weight regularisation, and the architectures, such as the block-based network topology and the parameters of each CNN block. The experimental results on five intrusion detection datasets in IIoT, including secure water treatment, water distribution, Gas Pipeline, Botnet in Internet of Things and Power System Attack Dataset, have demonstrated the superiority of the proposed SOPA-GA-CNN to the state-of-the-art manually designed models and neuron-evolutionary methods in terms of accuracy, precision, recall, F1score, and the number of parameters of the deep learning models.

show abstract

Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Cited by 222 publications

References 114 publications

A comparative patient-level prediction study in OMOP CDM demonstrates advantages for native and web-based implementations

A comparative patient-level prediction study in OMOP CDM demonstrates advantages for native and web-based implementations

General Cyclical Training of Neural Networks

SOPA‐GA‐CNN: Synchronous optimisation of parameters and architectures by genetic algorithms with convolutional neural network blocks for securing Industrial Internet‐of‐Things

Contact Info

Product

Resources

About