VolcanoML: speeding up end-to-end AutoML via scalable search space decomposition

Li, Yang; Shen, Yu; Zhang, Wentao; Zhang, Ce; Cui, Bin

doi:10.1007/s00778-022-00752-2

Cited by 18 publications

(12 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, there is a large effort in the data management community to speed up AutoML systems. For instance, Li et al propose to leverage search space decomposition [29]. Yakovlev et al propose to leverage proxy models, iteration-free optimization, and adaptive data reduction to accelerate hyperparameter optimization [54].…”

Section: Related Workmentioning

confidence: 99%

Data Cleaning and AutoML: Would an Optimizer Choose to Clean?

et al. 2022

View full text Add to dashboard Cite

Data cleaning is widely acknowledged as an important yet tedious task when dealing with large amounts of data. Thus, there is always a cost-benefit trade-off to consider. In particular, it is important to assess this trade-off when not every data point and data error is equally important for a task. This is often the case when statistical analysis or machine learning (ML) models derive knowledge about data. If we only care about maximizing the utility score of the applications, such as accuracy or F1 scores, many tasks can afford some degree of data quality problems. Recent studies analyzed the impact of various data error types on vanilla ML tasks, showing that missing values and outliers significantly impact the outcome of such models. In this paper, we expand the setting to one where data cleaning is not considered in isolation but as an equal parameter among many other hyper-parameters that influence feature selection, regularization, and model selection. In particular, we use state-of-the-art AutoML frameworks to automatically learn the parameters that benefit a particular ML binary classification task. In our study, we see that specific cleaning routines still play a significant role but can also be entirely avoided if the choice of a specific model or the filtering of specific features diminishes the overall impact.

show abstract

Section: Related Workmentioning

confidence: 99%

Data Cleaning and AutoML: Would an Optimizer Choose to Clean?

et al. 2022

View full text Add to dashboard Cite

show abstract

“…There are ongoing efforts to increase the performance of AutoML tools through better NAS approach [45,62,80,86], search for hyperparameters and loss methods [72,74,74], automating machine learning via keeping human in the loop [71]. Truong et al [104] evaluated few popular AutoML tools on their abilities to automate ML pipeline.…”

Section: Related Workmentioning

confidence: 99%

Challenges and Barriers of Using Low Code Software for Machine Learning

Alamin¹,

Uddin²

2022

Preprint

View full text Add to dashboard Cite

As big data grows ubiquitous across many domains, more and more stakeholders seek to develop Machine Learning (ML) applications on their data. The success of an ML application usually depends on the close collaboration of ML experts and domain experts. However, the shortage of ML engineers remains a fundamental problem. Low-code Machine learning tools/platforms (aka, AutoML) aim to democratize ML development to domain experts by automating many repetitive tasks in the ML pipeline, such as data collection, data pre-processing, feature engineering, model design, optimal hyper-parameter configuration, and model evaluation. However, even with minimal the hand coding support via the end-to-end pipeline of AutoML tools, it still requires human involvement in vital steps such as understanding the problem scope, domain-specific data, designing an appropriate training and testing dataset, model deployment & monitoring. As AutoML has some unique characteristics to traditional ML, it is vital to study the challenges ML practitioners face while they use the currently available AutoML tools, so that appropriate measures can be taken to address the challenges and to realize the vision of democratizing ML using low code software development principles and methodologies.This research presents an empirical study of around 14k posts (questions + accepted answers) from Stack Overflow (SO) that contained AutoML-related discussions. Software developers frequently use the online developer Q&A site SO to seek technical assistance. We observe a growing number of AutoML-related discussions in SO. We use LDA Topic Modeling to determine the topics discussed in those posts. Additionally, we examine how these topics are spread across the various Machine Learning Life Cycle (MLLC) phases and their popularity and difficulty. This study offers several interesting findings. First, we find 13 AutoML topics that we group into four categories. The MLOps topic category (43% questions) is the largest, followed by Model (28% questions), Data (27% questions), Documentation (2% questions). Second, Most questions are asked during Model training (29%) (i.e., implementation phase) and Data preparation (25%) MLLC phase. Third, AutoML practitioners find the MLOps topic category most challenging, especially topics related to model deployment & monitoring and Automated ML pipeline. Fourth, the Requirement analysis and scope definition MLLC phase are the most popular and challenging for AutoML practitioners. They also find the Model deployment and Model evaluation MLLC phase more complex than other phases like Data preparation and Model training. Fifth, MLOps topic category and Model deployment and Monitoring phase are more predominant and popular in cloud-based AutoML solution and Model topic category whereas Model evaluation phase is more dominant and popular in non-cloud AutoML solutions. These findings have implications for all three AutoML stakeholders: AutoML researchers, AutoML service vendors, and AutoML developers. Academia and Industry collaboration c...

show abstract

“…AutoCTS is a gradient-based method due to its high efficiency. Despite of great success in computer vision [3,37], natural language processing [39], and AutoML systems [27,54], little effort has been devoted to time series forecasting. AutoST [25] is proposed for spatio-temporal prediction, where the time series are from a uniform grid.…”

Section: Related Workmentioning

confidence: 99%

AutoCTS

Zhang

Guo

et al. 2021

Proc. VLDB Endow.

View full text Add to dashboard Cite

Correlated time series (CTS) forecasting plays an essential role in many cyber-physical systems, where multiple sensors emit time series that capture interconnected processes. Solutions based on deep learning that deliver state-of-the-art CTS forecasting performance employ a variety of spatio-temporal (ST) blocks that are able to model temporal dependencies and spatial correlations among time series. However, two challenges remain. First, ST-blocks are designed manually, which is time consuming and costly. Second, existing forecasting models simply stack the same ST-blocks multiple times, which limits the model potential. To address these challenges, we propose AutoCTS that is able to automatically identify highly competitive ST-blocks as well as forecasting models with heterogeneous ST-blocks connected using diverse topologies, as opposed to the same ST-blocks connected using simple stacking. Specifically, we design both a micro and a macro search space to model possible architectures of ST-blocks and the connections among heterogeneous ST-blocks, and we provide a search strategy that is able to jointly explore the search spaces to identify optimal forecasting models. Extensive experiments on eight commonly used CTS forecasting benchmark datasets justify our design choices and demonstrate that AutoCTS is capable of automatically discovering forecasting models that outperform state-of-the-art human-designed models.

show abstract

VolcanoML: speeding up end-to-end AutoML via scalable search space decomposition

Cited by 18 publications

References 50 publications

Data Cleaning and AutoML: Would an Optimizer Choose to Clean?

Data Cleaning and AutoML: Would an Optimizer Choose to Clean?

Challenges and Barriers of Using Low Code Software for Machine Learning

AutoCTS

Contact Info

Product

Resources

About