A practical evaluation of AutoML tools for binary, multiclass, and multilabel classification

Abstract

Choosing the right Automated Machine Learning (AutoML) tool is crucial for researchers of varying expertise to achieve optimal performance in diverse classification tasks. However, the abundance of AutoML frameworks with varying features makes selection challenging. This study addresses this gap by conducting a practical evaluation informed by a theoretical and bibliographical review and a feature-based comparison of twelve AutoML frameworks. The evaluation, conducted under time constraints, assessed accuracy and training efficiency across binary, multiclass, and multilabel (considering both native and label powerset representations) classification tasks on fifteen datasets. We acknowledge limitations, including dataset scope and default parameter usage, which may not capture the full potential of some frameworks. Our findings reveal no single ``perfect'' tool, as frameworks prioritize accuracy or speed. For time-sensitive binary/multiclass tasks, \claas, \autogluon, and \autokeras showed promise. In multilabel scenarios, \autosklearn offered higher accuracy, while \autokeras excelled in training speed. These results highlight the crucial trade-off between accuracy and speed, emphasizing the importance of considering both factors during tool selection for binary, multiclass, and multilabel classification problems. We made the code, experiment reproduction instructions, and outcomes publicly available on GitHub.

Enhancing AutoML Performance for Imbalanced Tabular Data Classification: A Self-Balancing Pipeline

“…The datasets used in the experiments were the same employed in Aragão et al (2023). It was done once the presented datasets contained the necessary properties to apply to the current study and enable future comparisons between both studies.…”

“…The hill-valley (1479) dataset is perfectly balanced once both classes have the same number of samples. This dataset was kept in the study since this feature must be lost in the data split step (described ahead) and to be compared with (Aragão et al, 2023). On the other hand, diabetes (37) and wdbc (1510) are the most imbalanced binary datasets.…”

“…It was one of the AutoML frameworks compared by (Aragão et al, 2023), and the combination of high scoring and execution speed in binary and multiclass scenarios made it eligible to be the evaluation framework implemented on the balancing pipeline in this study.…”

“…Before the grid search, it is done to ensure the evaluation is applied to the same data set. The data split was not stratified to keep the results comparable to Aragão et al (2023) once it was not done there. It may lead to subtle changes in the classes' original proportions; • Oversampling: takes the input dataset, calculates how many records of each class must be generated due to a decision function, and applies the resample according to the selected method and the selected number of records; • Undersampling: similar to the oversampling step, takes the input dataset, calculates how many records of each class must be removed due to a decision function, and applies the resample according to the selected method and the selected number of records; and • AutoML evaluation: uses the dataset processed by the above steps to create an AutoML model and evaluates it using the test dataset.…”

“…The code was written in Python, and its version was fixed as 3.8.18 to be consistent with Aragão et al (2023).…”