Stream-based active learning for sliding windows under the influence of verification latency

Abstract

Stream-based active learning (AL) strategies minimize the labeling effort by querying labels that improve the classifier’s performance the most. So far, these strategies neglect the fact that an oracle or expert requires time to provide a queried label. We show that existing AL methods deteriorate or even fail under the influence of such verification latency. The problem with these methods is that they estimate a label’s utility on the currently available labeled data. However, when this label would arrive, some of the current data may have gotten outdated and new labels have arrived. In this article, we propose to simulate the available data at the time when the label would arrive. Therefore, our method Forgetting and Simulating (FS) forgets outdated information and simulates the delayed labels to get more realistic utility estimates. We assume to know the label’s arrival date a priori and the classifier’s training data to be bounded by a sliding window. Our extensive experiments show that FS improves stream-based AL strategies in settings with both, constant and variable verification latency.

Stream-based Active Learning with Verification Latency in Non-stationary Environments

“…Verification latency was introduced in [14] considering three different variants: null latency refers to a label for a selected sample arriving instantaneously -this is the most common setting for AL in the literature; extreme latency, i.e. the label is never available -this setting is getting a lot of attention recently [21]; and intermediate latency, with a finite delay between sample selection and label arrival -this is common in many real-world applications, but has received only little attention in the literature [18]. In [10], the authors list research questions and approaches addressing verification latency in streaming data, but do not consider AL strategies.…”

“…AL strategies are considered in [13,17,19], where delayed labels are directly incorporated to the training loop of various models. A utility estimation method similar to the one proposed here is considered in [18] which addresses the effect of delayed labels. Yet the work makes unrealistic assumptions about the label delay, e.g., fixed delay which is also known a priori; the authors point out the necessity for more research on generalizations thereof [18,20].…”

“…A utility estimation method similar to the one proposed here is considered in [18] which addresses the effect of delayed labels. Yet the work makes unrealistic assumptions about the label delay, e.g., fixed delay which is also known a priori; the authors point out the necessity for more research on generalizations thereof [18,20]. Drift detection in presence of verification latency is getting some attention in recent years [6,16].…”

“…The presence of verification latency can be particularly problematic for machine learning approaches which deal with streaming data subject to concept drift, because a label might already be wrong when it arrives. To the best of our knowledge, the effect of unknown verification latency on AL strategies is currently widely unexplored [18]. Further, most AL strategies assume a homogeneous budget over time, albeit label requests might be particularly beneficial right after a detected drift event [12,24].…”

“…In the following, we investigate the performance of unsupervised and semi-supervised drift detectors under the effect of verification latency. We test the performance by a thorough evaluation of several synthetic and real-world non-stationary datasets, with several realistic settings of verification delay, and we compare the proposed strategies against the State-Of-The-Art (SOTA) [18].…”