Fitness Landscape Analysis of Automated Machine Learning Search Spaces

Abstract

The field of Automated Machine Learning (AutoML) has as its main goal to automate the process of creating complete Machine Learning (ML) pipelines to any dataset without requiring deep user expertise in ML. Several AutoML methods have been proposed so far, but there is not a single one that really stands out. Furthermore, there is a lack of studies on the characteristics of the fitness landscape of AutoML search spaces. Such analysis may help to understand the performance of different optimization methods for AutoML and how to improve them. This paper adapts classic fitness landscape analysis measures to the context of AutoML. This is a challenging task, as AutoML search spaces include discrete, continuous, categorical and conditional hyperparameters. We propose an ML pipeline representation, a neighborhood definition and a distance metric between pipelines, and use them in the evaluation of the fitness distance correlation (FDC) and the neutrality ratio for a given AutoML search space. Results of FDC are counter-intuitive and require a more in-depth analysis of a range of search spaces. Results of neutrality, in turn, show a strong positive correlation between the mean neutrality ratio and the fitness value.

Neural Architecture Search in Graph Neural Networks

“…It may seem counter-intuitive that we are using sophisticated methods to obtain results that can be also be achieved by a random search method, but as authors in [2] have previously discussed, in large search spaces where many of the dimensions are irrelevant to the task at hand the random search can be as effective as more sophisticated methods. This problem is aggravated by the neutrality of the space, i.e., architectures in neighbour regions of the search space may differ in a few components but do not lead to a value of accuracy different from their neighbors [15]. Another stronger indicator of a neutral search space is the fact that many high quality individuals are generated in the initialization step, and evolution takes a minor part in improving them, as shown in Figure 4.…”

A Survey of Advances in Landscape Analysis for Optimisation

“…Since 2017 a new trend has emerged of landscape analysis applied in the context of machine learning. Examples include: analysis of weight search spaces in the context of neural network training for classification [98]; analysis of the feature selection problem for classification [99,100]; analysis of policy search spaces in reinforcement learning [101]; analysis of machine learning pipeline configuration search spaces [102]; and analysis of neural architecture search spaces for image classification [103,104].…”

Searching in the Forest for Local Bayesian Optimization

“…However, previous work on HPO showed two important insights: First, the loss landscape of a hyperparameter optimization problem is more benign than what one would expect (Klein et al, 2017;Pushak and Hoos, 2018;Pimenta et al, 2020). In most cases, the loss landscape in well-performing regions is quite flat and the best performing region is fairly well defined, see for example Figure 1.…”