Explorative data analysis of time series based algorithm features of CMA-ES variants

Abstract

In this study, we analyze behaviours of the well-known CMA-ES by extracting the time-series features on its dynamic strategy parameters. An extensive experiment was conducted on twelve CMA-ES variants and 24 test problems taken from the BBOB (Black-Box Optimization Bench-marking) testbed, where we used two different cutoff times to stop those variants. We utilized the tsfresh package for extracting the features and performed the feature selection procedure using the Boruta algorithm, resulting in 32 features to distinguish either CMA-ES variants or the problems. After measuring the number of predefined targets reached by those variants, we contrive to predict those measured values on each test problem using the feature. From our analysis, we saw that the features can classify the CMA-ES variants, or the function groups decently, and show a potential for predicting the performance of those variants. We conducted a hierarchical clustering analysis on the test problems and noticed a drastic change in the clustering outcome when comparing the longer cutoff time to the shorter one, indicating a huge change in search behaviour of the algorithm. In general, we found that with longer time series, the predictive power of the time series features increase. CCS CONCEPTS• Theory of computation → Bio-inspired optimization; • Mathematics of computing → Time series analysis; Exploratory data analysis.

Trajectory-based Algorithm Selection with Warm-starting

“…We believe that the regression models can strongly benefit from this information; possibly not in the naïve way applied in [5] (where only the final state variables at the time of the switch were used as features for the regression model), but by extracting information from the evolution of the state variables during the first part of the optimization process, before the switch. Such an approach based on time-series analysis have been suggested in the literature [44]. There, it was shown that features computed on evolution of the state variables of the CMA-ES can be used to accurately classify variants of the algorithm, and predict which of the BBOB problems was being optimized.…”

Trajectory-based Algorithm Selection with Warm-starting

“…We believe that the regression models can strongly benefit from this information; possibly not in the naïve way applied in [5] (where only the final state variables at the time of the switch were used as features for the regression model), but by extracting information from the evolution of the state variables during the first part of the optimization process, before the switch. Such an approach based on time-series analysis have been suggested in the literature [43]. There, it was shown that features computed on evolution of the state variables of the CMA-ES can be used to accurately classify variants of the algorithm, and predict which of the BBOB problems was being optimized.…”

The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants

“…Several studies explored the family of modular CMA-ES in different learning settings including exploratory analysis of CMA-ES [9], automated algorithm performance prediction [31], automated algorithm selection [16], and automated algorithm configuration [5,29].…”

“…De Nobel et al [9] analyzed the CMA-ES behavior by using timeseries features extracted from its dynamic strategy parameters. Their results showed that these features can be used to classify isolated CMA-ES modules and have the potential to be used in the prediction of their performance.…”

“…That is, they do not aim at exploring the impact that each module has on the final performance of the CMA-ES configuration. In [9], the time-series features have been used to classify only the isolated CMA-ES modules, but there is no information on how these features are linked when those modules are combined to create a new CMA-ES configuration. Prager et al [29] investigated a problem instance-based configuration model that selects the optimal CMA-ES modules to create a configuration specific to a given problem instance.…”