Change with Delayed Labeling: When is it Detectable?

liobaite, Indre

doi:10.1109/icdmw.2010.49

Cited by 74 publications

(72 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After drift happens, the labeling information of recent data samples is used to retrain the classifier. However, those methods in [5]- [7] can only be applied for binary class problems.…”

Section: Related Workmentioning

confidence: 99%

“…For the unlabeled data, the reference window and the detection window's confidence distributions are used to detect the change. The method in [5] analyzes a sequence of the posterior estimates derived from the classifier by using the univariate statistical tests such as Two sample t-test and Wilcoxon Rank sum test. CDBD (Confidence Distribution Batch Detection) [6] uses the confidence estimated by the classifier.…”

Section: Related Workmentioning

confidence: 99%

“…However, the assumption that labels become available immediately after a data sample arrives is not generally suitable when processing the data stream due to the labeling costs and the time-consuming process to obtain labels. When access to the labels is limited and only small amount of labeled data is available, concept drift can be detected by monitoring the classifier output [5]- [7]. With no access to the labels of data samples, there is no information about the labels of the data stream.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Efficient Concept Drift Detection Method for Streaming Data under Limited Labeling

Kim

Park

2017

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

“…After drift happens, the labeling information of recent data samples is used to retrain the classifier. However, those methods in [5]- [7] can only be applied for binary class problems.…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Efficient Concept Drift Detection Method for Streaming Data under Limited Labeling

Kim

Park

2017

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

“…The detector can be based on changes in the probability distribution of the instances (Gaber and Yu 2006;Markou and Singh 2003) or classification accuracy (Klinkenberg and Joachims 2000;Baena-García 2006). Many detection algorithms are based on a knowledge of object labels after the classification to detect the presence of a concept drift, however as pointed out in Zliobaite (2010), such approach is not useful from a practical point of view.…”

Section: Related Workmentioning

confidence: 99%

“…Some evolving systems continuously adjust the model to incoming data (Zliobaite 2010), what is called implicit drift detection (Kuncheva 2008) as opposed to explicit drift detection methods that raise a signal to indicate change. The detector can be based on changes in the probability distribution of the instances (Gaber and Yu 2006;Markou and Singh 2003) or classification accuracy (Klinkenberg and Joachims 2000;Baena-García 2006).…”

Section: Related Workmentioning

confidence: 99%

One-class classifiers with incremental learning and forgetting for data streams with concept drift

Krawczyk

Woźniak

2014

Soft Comput

View full text Add to dashboard Cite

One of the most important challenges for machine learning community is to develop efficient classifiers which are able to cope with data streams, especially with the presence of the so-called concept drift. This phenomenon is responsible for the change of classification task characteristics, and poses a challenge for the learning model to adapt itself to the current state of the environment. So there is a strong belief that one-class classification is a promising research direction for data stream analysis-it can be used for binary classification without an access to counterexamples, decomposing a multi-class data stream, outlier detection or novel class recognition. This paper reports a novel modification of weighted one-class support vector machine, adapted to the non-stationary streaming data analysis. Our proposition can deal with the gradual concept drift, as the introduced one-class classifier model can adapt its decision boundary to new, incoming data and additionally employs a forgetting mechanism which boosts the ability of the classifier to follow the model changes. In this work, we propose several different strategies for incremental learning and forgetting, and additionally we evaluate them on the basis of several real data streams. Obtained results confirmed the usability of proposed classifier to the problem of data stream classification with the presence of concept drift. Additionally, implemented

show abstract

KS(conf): A Light-Weight Test if a ConvNet Operates Outside of Its Specifications

Sun

Lampert

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Computer vision systems for automatic image categorization have become accurate and reliable enough that they can run continuously for days or even years as components of real-world commercial applications. A major open problem in this context, however, is quality control. Good classification performance can only be expected if systems run under the specific conditions, in particular data distributions, that they were trained for. Surprisingly, none of the currently used deep network architectures has a built-in functionality that could detect if a network operates on data from a distribution that it was not trained for and potentially trigger a warning to the human users.In this work, we describe KS(conf), a procedure for detecting such outside of the specifications operation. Building on statistical insights, its main step is the applications of a classical Kolmogorov-Smirnov test to the distribution of predicted confidence values. We show by extensive experiments using Im-ageNet, AwA2 and DAVIS data on a variety of ConvNets architectures that KS(conf) reliably detects out-of-specs situations. It furthermore has a number of properties that make it an excellent candidate for practical deployment: it is easy to implement, adds almost no overhead to the system, works with all networks, including pretrained ones, and requires no a priori knowledge about how the data distribution could change.

show abstract

Change with Delayed Labeling: When is it Detectable?

Cited by 74 publications

References 22 publications

An Efficient Concept Drift Detection Method for Streaming Data under Limited Labeling

An Efficient Concept Drift Detection Method for Streaming Data under Limited Labeling

One-class classifiers with incremental learning and forgetting for data streams with concept drift

KS(conf): A Light-Weight Test if a ConvNet Operates Outside of Its Specifications

Contact Info

Product

Resources

About