Suitability of Different Metric Choices for Concept Drift Detection

Abstract

The notion of concept drift refers to the phenomenon that the distribution, which is underlying the observed data, changes over time; as a consequence machine learning models may become inaccurate and need adjustment. Many unsupervised approaches for drift detection rely on measuring the discrepancy between the sample distributions of two time windows. This may be done directly, after some preprocessing (feature extraction, embedding into a latent space, etc.), or with respect to inferred features (mean, variance, conditional probabilities etc.). Most drift detection methods can be distinguished in what metric they use, how this metric is estimated, and how the decision threshold is found. In this paper, we analyze structural properties of the drift induced signals in the context of different metrics. We compare different types of estimators and metrics theoretically and empirically and investigate the relevance of the single metric components. In addition, we propose new choices and demonstrate their suitability in several experiments. * We gratefully acknowledge funding by the BMBF TiM, grant number 05M20PBA.

Precise Change Point Detection using Spectral Drift Detection

“…MomentTree-Kernel Although any kernel can be used for SDDM, we suggest to make use of a kernel based on MomentTrees [18] which seems to provide a better distance measure than the commonly used Gauss-kernel [19].…”

“…MomentTrees is a method for decision tree based conditional density estimation; in the context of drift they are trained to predict the arrival time t based on the sample x [19], i.e. they learn x → t. The idea is derived from the observation that, by using conditional density estimation, the data space can Algorithm 1 Spectral Drift Detection Method 1: procedure SDDM: Spectral Drift Detection Method((x i ) data stream, n eigen number of eigenvectors, n itr number of iterations in crossvalidation, k max maximal number of drifts)…”

“…end while 21: end procedure be segmented into regions with the same drift behavior [17]. Due to this fact, the trees will find a segmentation of the data space that provides a particularly good binning to measure the distance between p t and p s [19].…”