A Ranking Stability Measure for Quantifying the Robustness of Anomaly Detection Methods

Abstract

Anomaly detection attempts to learn models from data that can detect anomalous examples in the data. However, naturally occurring variations in the data impact the model that is learned and thus which examples it will predict to be anomalies. Ideally, an anomaly detection method should be robust to such small changes in the data. Hence, this paper introduces a ranking stability measure that quantifies the robustness of any anomaly detector's predictions by looking at how consistently it ranks examples in terms of their anomalousness. Our experiments investigate the performance of this stability measure under different data perturbation schemes. In addition, they show how the stability measure can complement traditional anomaly detection performance measures, such as area under the ROC curve or average precision, to quantify the behaviour of different anomaly detection methods.

Transferring the Contamination Factor between Anomaly Detection Domains by Shape Similarity

“…Although calibration usually requires either labeled examples or a known contamination factor, Kriegel et al (2011) introduce UNIFY, a method to obtain calibrated probabilities from anomaly scores without such requirements. In absence of labeled data, Marques et al (2020) develop an internal measure to evaluate the quality of an anomaly detector, while Schubert et al (2012) and Perini et al (2020) develop rank similarity measures to compare the anomaly rankings of different detectors. However, none of these works propose a method to find an appropriate decision threshold for the anomaly scores in an (unlabeled) dataset.…”