Classifier self-assessment: active learning and active noise correction for document classification

“…In fact, effectively utilizing expert oracles to bootstrap an AI system has become a common challenge for enterprise AI applications. Actively denoising labeled data is not a new topic [10,12,14,20,21,24,27,29]. The problem was studied under the inductive logic programming framework in the early days of Machine Learning.…”

Section: Problem Statement and Related Workmentioning

confidence: 99%

Active Deep Learning to Tune Down the Noise in Labels

Samel

2018

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

The great success of supervised learning has initiated a paradigm shift from building a deterministic software system to a probabilistic artificial intelligent system throughout the industry. The historical records in enterprise domains can potentially bootstrap the traditional business into the modern data-driven approach almost everywhere. The introduction of the Deep Neural Networks (DNNs) significantly reduces the efforts of feature engineering so that supervised learning becomes even more automated. The last bottleneck is to ensure the data quality, particularly the label quality, because the performance of supervised learning is bounded by the errors present in labels. In this paper, we present a new Active Deep Denoising (ADD) approach that first builds a DNN noise model, and then adopts an active learning algorithm to identify the optimal denoising function. We prove that under the low noise condition, we only need to query the oracle with log n examples where n is the total number in the data. We apply ADD on one enterprise application and show that it can effectively reduce 1 3 of the prediction error with only 0.1% of examples verified by the oracle.

show abstract

“…2. Margin based mislabeling measure [12,4]. This measure simply suggest that a mislabeled instance is the one having a high prediction probability and a low probability for the label given by the oracle.…”

Section: Entropy Reduction Based Mislabeling Measure [33]mentioning

confidence: 99%

“…A strategy proposed in [20] rely on the classifier's confidence to actively ask the correction of the suspected (i.e., possibly mislabeled) instances from an expert, however, the learning in itself is passive. The same strategy has been investigated for active learning in [12] and [4], where suspected instances can be relabeled or discarded. The active learning method proposed in [33], suggests that a suspiciously mislabeled instance is the one that minimizes the expected entropy over the unlabeled dataset if it is labeled with a new label other than the one given by the oracle.…”

Section: Introductionmentioning

confidence: 99%