Novel Meta-Features for Automated Machine Learning Model Selection in Anomaly Detection

Abstract

A growing number of research papers shed light on automated machine learning (AutoML) frameworks, which are becoming a promising solution for building complex machine learning models without human expertise and assistance. The key challenge in enabling AutoML frameworks to build an efficient model for anomaly detection tasks is to determine the best underlying model for a given task and optimization metric. The meta-learning approaches based on a set of meta-features that describes data properties can enable efficient model selection in AutoML frameworks. The existing meta-learning approaches based on statistical and information-theoretic meta-features require large amounts of data and computational resources to extract data properties. This paper proposes a novel set of meta-features for model selection in anomaly detection tasks based on domain-specific properties of data which overcomes the shortcomings of existing meta-features by introducing simple but effective meta-features that can be efficiently extracted or estimated by using a low amount of data. Experiments with 63 datasets from different repositories with varying schemas show that the proposed set of meta-features achieves an accuracy of 87% for model selection, while the achieved accuracy for simple meta-features is 74%, for statistical meta-features 68%, for information theory meta-feature 70%, and for a comprehensive set of meta-features by pyMFE 73%. This demonstrates that the proposed set can be adopted by AutoML frameworks across a diverse range of domains.

Constructing a Meta-Learner for Unsupervised Anomaly Detection

“…This outcome is an important consideration given that current AutoML or meta-learning studies frequently direct their attention to other aspects, such as meta-features development [19], [31] or HO [23], [24], [26]. This work demonstrates that investing time and effort into creating an adequate meta-model that can successfully utilise data from historical evaluations is the most promising approach for improving meta-learners for unsupervised AD.…”

“…Few studies have proposed an automated algorithm selection in AD [17], [18], [19], [20], [21]. Of these, only one model is designed for unsupervised scenarios and utilises the meta-learner framework as presented by Rice [18].…”

“…The ASP for unsupervised settings remains mostly unexplored. A few studies have examined the meta-features' potential in AD scenarios [1], [18], [19], [43], but their limitations are either due to the required knowledge of data labels or a structure, or a need for an extensive evaluation of algorithms for a new task. The approach presented in [18] has addressed the ASP problem for unsupervised tasks by utilising the knowledge gained through past algorithm evaluation and is the state of the art solution for the aforementioned problem.…”

“…An alternative set of meta-features for AD tasks has been proposed by Kotlar et al [19]. Their research focused on semantic features designed explicitly for AD, such as global, local, or collective anomaly types, as well as other characteristics like anomaly space, anomaly ratio, data type, or data domain.…”

Metamodelling of Noise to Image Classification Performance

“…Additionally, they designed a methodology where the relation between dataset characteristics and objective performance measures is captured in a regression-based metamodel. Kotlar et al [19] elaborate on the performance of anomaly detectors on different datasets. They capture the relationship between characteristics of the anomaly-containing datasets and the performance of detectors into a regression-based metamodel.…”