A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping

Abstract

A survey of existing methods for stopping active learning (AL) reveals the needs for methods that are: more widely applicable; more aggressive in saving annotations; and more stable across changing datasets. A new method for stopping AL based on stabilizing predictions is presented that addresses these needs. Furthermore, stopping methods are required to handle a broad range of different annotation/performance tradeoff valuations. Despite this, the existing body of work is dominated by conservative methods with little (if any) attention paid to providing users with control over the behavior of stopping methods. The proposed method is shown to fill a gap in the level of aggressiveness available for stopping AL and supports providing users with control over stopping behavior.

Batch Mode Active Learning for Multimedia Pattern Recognition

“…Overfitting may be one possible reason to explain this observation. There have been some efforts to address this problem [325,326,327]. These methods are fairly similar, based on the notion of an intrinsic measure of stability or self-confidence of the learner and that active learning ceases to be useful once this measure begins to degrade.…”

Active Learning with Subsequence Sampling Strategy for Sequence Labeling Tasks

“…S t and S t+1 ), we can stop the learning. We measure the difference of predicted sets using Kappa statistics (Bloodgood and Vijay-Shanker 2009). From our experiment, standard κ threshold at 0.99 makes the learning stop so early that the system cannot achieve high F 1 .…”

Identifying PCB Contaminated Transformers Through Active Learning

“…In addition to the confidence based stopping criteria, Bloodgood and Vijay-Shanker [4] proposed a stability based stopping criteria. Bloodgood and Vijay-Shanker consider stability as a necessary factor to determine when the active learner should terminate.…”