Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Zimmer, Lucas; Lindauer, Marius; Hutter, Frank

doi:10.48550/arxiv.2006.13799

Cited by 5 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different settings beyond supervised learning have been investigated in NAS, including like semi-supervised learning [392], self-supervised learning [131], unsupervised learning [115], [377], incremental learning [361], [393], federated learning [394], [395], etc., showing the promising transferability of NAS methods. Last but not least, there are several toolkits for AutoML [17], [396], [397], [398], [399] that can facilitate the reproducibility of NAS methods.…”

Section: Discussionmentioning

confidence: 99%

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Xie¹,

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. In the early age, researchers mostly applied individual search methods which sample and evaluate the candidate architectures separately and thus incur heavy computational overheads. To alleviate the burden, weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network, and the costly training procedure is performed only once. These methods, though being much faster, often suffer the issue of instability. This paper provides a literature review on NAS, in particular the weight-sharing methods, and points out that the major challenge comes from the optimization gap between the super-network and the sub-architectures. From this perspective, we summarize existing approaches into several categories according to their efforts in bridging the gap, and analyze both advantages and disadvantages of these methodologies. Finally, we share our opinions on the future directions of NAS and AutoML. Due to the expertise of the authors, this paper mainly focuses on the application of NAS to computer vision problems and may bias towards the work in our group.

show abstract

Section: Discussionmentioning

confidence: 99%

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Xie¹,

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, TPOT from [29] generates a set of best performing models from Sklearn and XGboost and automatically chose the best subset of models. Moreover, other papers focus on the problem of automated deep learning model selection and optimization [24,43]. Finally, several papers propose various methods of automatic feature generation [29].…”

Section: Related Workmentioning

confidence: 99%

LightAutoML: AutoML Solution for a Large Financial Services Ecosystem

Vakhrushev¹,

Ryzhkov²,

Savchenko³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present an AutoML system called LightAutoML developed for a large European financial services company and its ecosystem satisfying the set of idiosyncratic requirements that this ecosystem has for AutoML solutions. Our framework was piloted and deployed in numerous applications and performed at the level of the experienced data scientists while building high-quality ML models significantly faster than these data scientists. We also compare the performance of our system with various general-purpose open source AutoML solutions and show that it performs better for most of the ecosystem and OpenML problems. We also present the lessons that we learned while developing the AutoML system and moving it into production.

show abstract

“…The nodes N 1 and N 2 represent dense layers Dense(x, y), where x is the number of neurons and y is the activation function. The nodes SC 2 1 , SC 3 1 , SC 3 2 represent the possible skip-connection nodes, when id R is chosen for each of them. The node N 2 is connected to input node through SC 2 1 .…”

Section: Repeatmentioning

confidence: 99%

“…The nodes SC 2 1 , SC 3 1 , SC 3 2 represent the possible skip-connection nodes, when id R is chosen for each of them. The node N 2 is connected to input node through SC 2 1 . The output node is connected to input and N 1 nodes through SC 3 1 and SC 3 2 , respectively.…”

Section: Repeatmentioning

confidence: 99%

“…Tabular data sets are often diverse. They are obtained from multiple sources and modes, where combining certain inputs using problem-specific domain knowledge typically leads to better and physically meaningful features and consequently robust models [1], [2]. Many high-performing predictive mod-els for tabular data are based on classical supervised machine learning (ML) methods such as bagging, boosting, and kernelbased methods.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data

Égelé¹,

Balaprakash²,

Vishwanath³

et al. 2020

Preprint

View full text Add to dashboard Cite

Developing high-performing predictive models for large tabular data sets is a challenging task. The state-of-theart methods are based on expert-developed model ensembles from different supervised learning methods. Recently, automated machine learning (AutoML) is emerging as a promising approach to automate predictive model development. Neural architecture search (NAS) is an AutoML approach that generates and evaluates multiple neural network architectures concurrently and improves the accuracy of the generated models iteratively. A key issue in NAS, particularly for large data sets, is the large computation time required to evaluate each generated architecture. While data-parallel training is a promising approach that can address this issue, its use within NAS is difficult. For different data sets, the data-parallel training settings such as the number of parallel processes, learning rate, and batch size need to be adapted to achieve high accuracy and reduction in training time.To that end, we have developed AgEBO-Tabular, an approach to combine aging evolution (AgE), a parallel NAS method that searches over neural architecture space, and an asynchronous Bayesian optimization method for tuning the hyperparameters of the data-parallel training simultaneously. We demonstrate the efficacy of the proposed method to generate high-performing neural network models for large tabular benchmark data sets. Furthermore, we demonstrate that the automatically discovered neural network models using our method outperform the stateof-the-art AutoML ensemble models in inference speed by two orders of magnitude while reaching similar accuracy values.

show abstract

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Cited by 5 publications

References 23 publications

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

LightAutoML: AutoML Solution for a Large Financial Services Ecosystem

AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data

Contact Info

Product

Resources

About