Adaptation Strategies for Automated Machine Learning on Evolving Data

Çelik, Bilge Gökhan; Vanschoren, Joaquin

doi:10.1109/tpami.2021.3062900

Cited by 54 publications

(49 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Taking advantage of the recent wave of research in AutoML, an alternative approach to adaptation to changing environments was proposed in Martín Salvador et al (2016) where repeated automated deployment of Auto-WEKA for Multi-Component Predictive Systems (MCPS) to learn from new batches of data was used for life-long learning and the adaptation of complex MCPS when applied to changing streaming data from process industries. Celik and Vanschoren (2021) represent a development of this idea with the inclusion of the drift detection and the experimentation using several open source AutoML frameworks. An interesting approach closely tied with the Auto_Sklearn is described in Madrid et al (2019).…”

Section: Automated Machine Learning For Streaming Datamentioning

confidence: 99%

Automated adaptation strategies for stream learning

Bakirov

Fay²,

Gabrys

2021

Mach Learn

View full text Add to dashboard Cite

Automation of machine learning model development is increasingly becoming an established research area. While automated model selection and automated data pre-processing have been studied in depth, there is, however, a gap concerning automated model adaptation strategies when multiple strategies are available. Manually developing an adaptation strategy can be time consuming and costly. In this paper we address this issue by proposing the use of flexible adaptive mechanism deployment for automated development of adaptation strategies. Experimental results after using the proposed strategies with five adaptive algorithms on 36 datasets confirm their viability. These strategies achieve better or comparable performance to the custom adaptation strategies and the repeated deployment of any single adaptive mechanism.

show abstract

Section: Automated Machine Learning For Streaming Datamentioning

confidence: 99%

Automated adaptation strategies for stream learning

Bakirov

Fay²,

Gabrys

2021

Mach Learn

View full text Add to dashboard Cite

show abstract

“…This module allows the automatic training and tuning of classification and regression models within a user-specified time-limit [34]. Specifically, it performs a fast random search in combination with automated stacked ensembles, using random forests, gradient boosting machines, deep neural nets, or generalised linear models [35].…”

Section: Related Workmentioning

confidence: 99%

Hyperparameter self-tuning for data streams

et al. 2021

View full text Add to dashboard Cite

The number of Internet of Things devices generating data streams is expected to grow exponentially with the support of emergent technologies such as 5G networks. The online processing of these data streams therefore requires the design and development of suitable machine learning algorithms, able to learn online, as data is generated. Like their batch-learning counterparts, streambased learning algorithms require careful hyperparameter settings. However, this problem is exacerbated in online learning settings, especially with the occurrence of concept drifts, that frequently require the reconfiguration of hyperparameters. In this article, we present SSPT, an extension of the Self Parameter Tuning (SPT) optimisation algorithm for data streams. We apply the Nelder-Mead algorithm to dynamically-sized samples, converging to optimal settings in a single-pass over data, while using a relatively small number of hyperparameter configurations. In addition, our proposal automatically readjusts hyperparameters when concept drift occurs. To assess the effectiveness of SSPT, the algorithm is evaluated with three different machine learning problems: recommendation, regression, and classification. Experiments with well-known data sets show that the proposed algorithm can outperform previous hyperparameter tuning efforts by human experts. Results show that SSPT converges significantly faster and

show abstract

“…Some recent efforts have focused on the problem of AutoML for streaming applications, where the optimal algorithm and hyperparameters might change over time (e.g., Veloso et al 2018;Carnein et al 2020;Celik and Vanschoren 2020). Although this might be interesting for EarlyTSC, in this paper we consider only static problems.…”

Section: Automated Machine Learningmentioning

confidence: 99%

MultiETSC: automated machine learning for early time series classification

Ottervanger

Baratchi

Hoos

2021

Data Min Knowl Disc

View full text Add to dashboard Cite

Early time series classification (EarlyTSC) involves the prediction of a class label based on partial observation of a given time series. Most EarlyTSC algorithms consider the trade-off between accuracy and earliness as two competing objectives, using a single dedicated hyperparameter. To obtain insights into this trade-off requires finding a set of non-dominated (Pareto efficient) classifiers. So far, this has been approached through manual hyperparameter tuning. Since the trade-off hyperparameters only provide indirect control over the earliness-accuracy trade-off, manual tuning is tedious and tends to result in many sub-optimal hyperparameter settings. This complicates the search for optimal hyperparameter settings and forms a hurdle for the application of EarlyTSC to real-world problems. To address these issues, we propose an automated approach to hyperparameter tuning and algorithm selection for EarlyTSC, building on developments in the fast-moving research area known as automated machine learning (AutoML). To deal with the challenging task of optimising two conflicting objectives in early time series classification, we propose MultiETSC, a system for multi-objective algorithm selection and hyperparameter optimisation (MO-CASH) for EarlyTSC. MultiETSC can potentially leverage any existing or future EarlyTSC algorithm and produces a set of Pareto optimal algorithm configurations from which a user can choose a posteriori. As an additional benefit, our proposed framework can incorporate and leverage time-series classification algorithms not originally designed for EarlyTSC for improving performance on EarlyTSC; we demonstrate this property using a newly defined, “naïve” fixed-time algorithm. In an extensive empirical evaluation of our new approach on a benchmark of 115 data sets, we show that MultiETSC performs substantially better than baseline methods, ranking highest (avg. rank 1.98) compared to conceptually simpler single-algorithm (2.98) and single-objective alternatives (4.36).

show abstract

Adaptation Strategies for Automated Machine Learning on Evolving Data

Cited by 54 publications

References 32 publications

Automated adaptation strategies for stream learning

Automated adaptation strategies for stream learning

Hyperparameter self-tuning for data streams

MultiETSC: automated machine learning for early time series classification

Contact Info

Product

Resources

About