AutoML for Multi-Label Classification: Overview and Empirical Evaluation

Abstract

Automated machine learning (AutoML) supports the algorithmic construction and data-specific customization of machine learning pipelines, including the selection, combination, and parametrization of machine learning algorithms as main constituents. Generally speaking, AutoML approaches comprise two major components: a search space model and an optimizer for traversing the space. Recent approaches have shown impressive results in the realm of supervised learning, most notably (single-label) classification (SLC). Moreover, first attempts at extending these approaches towards multi-label classification (MLC) have been made. While the space of candidate pipelines is already huge in SLC, the complexity of the search space is raised to an even higher power in MLC. One may wonder, therefore, whether and to what extent optimizers established for SLC can scale to this increased complexity, and how they compare to each other. This paper makes the following contributions: First, we survey existing approaches to AutoML for MLC. Second, we augment these approaches with optimizers not previously tried for MLC. Third, we propose a benchmarking framework that supports a fair and systematic comparison. Fourth, we conduct an extensive experimental study, evaluating the methods on a suite of MLC problems. We find a grammar-based best-first search to compare favorably to other optimizers.

AMLB: an AutoML Benchmark

“…A recent AutoML benchmark for multi-label classification Wever et al (2021) proposed a general tool with a configurable search space and optimizer, which allows for the inclusion of new methods and ablation studies. Unfortunately, this approach requires that existing AutoML frameworks be re-implemented within this tool, which is difficult in such a rapidly developing field.…”

Towards Green Automated Machine Learning: Status Quo and Future Directions

“…While Feurer et al [17] report computational cost of 11 CPU-years (CPUy), the experimental data of Mohr, Wever, and Hüllermeier [44] is the result of 52 CPUy worth of computations. In [77], the experimental study is as extensive as 84 CPUy. In the sub-field of neural architecture search (NAS) [13], computational costs for experimentation can even be higher.…”

“…Obviously, the energy consumption is relatively low, however, energy-efficiency in turn is poor since the information obtained through investing energy is not as valuable as desired. Various benchmarks for AutoML tools have already identified a plethora of confounding factors [3,21,77,86], which hinder interpretation of the results and insights derived from them.…”

Automated Machine Learning, Bounded Rationality, and Rational Metareasoning

“…In AutoML [16,11,44,48], the learner is no longer restricted to a pre-defined learning model (H, A), consisting of a hypothesis space and a learning algorithm. Instead, it can choose among a broad spectrum of learning algorithms A ∈ A, each associated with an underlying hypothesis space H A .…”