Deep Active Learning via Open-Set Recognition

Abstract

In many applications, data is easy to acquire but expensive and time-consuming to label, prominent examples include medical imaging and NLP. This disparity has only grown in recent years as our ability to collect data improves. Under these constraints, it makes sense to select only the most informative instances from the unlabeled pool and request an oracle (e.g., a human expert) to provide labels for those samples. The goal of active learning is to infer the informativeness of unlabeled samples so as to minimize the number of requests to the oracle. Here, we formulate active learning as an open-set recognition problem. In this paradigm, only some of the inputs belong to known classes; the classifier must identify the rest as unknown. More specifically, we leverage variational neural networks (VNNs), which produce high-confidence (i.e., low-entropy) predictions only for inputs that closely resemble the training data. We use the inverse of this confidence measure to select the samples that the oracle should label. Intuitively, unlabeled samples that the VNN is uncertain about contain features that the network has not been exposed to; thus they are more informative for future training. We carried out an extensive evaluation of our novel, probabilistic formulation of active learning, achieving state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and FashionMNIST. Additionally, unlike current active learning methods, our algorithm can learn even in the presence of out-of-distribution outliers. As our experiments show, when the unlabeled pool consists of a mixture of samples from multiple datasets, our approach can automatically distinguish between samples from seen vs. unseen datasets. Overall, our results show that high-quality uncertainty measures are key for pool-based active learning.

Automated labelling of radiology reports using natural language processing: Comparison of traditional and newer methods

“…Human‐labelled reports are important as ground truth for determining the performance of the various methods for automated labelling of radiology reports [ 12 ]. However, in radiology, as with all other medical fields, subject matter experts may be both difficult to find and expensive to engage [ 13 ]. To address this problem, some researchers have attempted to use weak supervision to label the data in place of humans [ 14 ].…”

A Cost-effective Deep Active Learning for Object Detection in Automated Driving Systems

“…The query-driven approaches were based on gaining support from complementary techniques to perform query improvement, such as optimization techniques, metrics learning [25], and alternative learning paradigms (one-shot, contrastive, federated, goal-driven, domain adaptive...) [26], [27], [28], [29], [30]. On the other side, data-driven approaches were attempted to address several data-level perspectives in terms of data labeling supervision (weak, self, semi...) [11], [31], [32], [33], labeling setting (open-set recognition) [34], [35] and granularity [18], [21]. For further details please refer to the survey papers [7], [8], [36].…”

Using dropout based active learning and surrogate models in the inverse viscoelastic parameter identification of human brain tissue

“…This is the aim of so called active learning approaches ( Hasenjäger and Ritter, 2002 ). Several approaches have been developed for image classification tasks, e.g., BatchBALD ( Kirsch et al, 2019 ), variational adversarial active learning ( Sinha et al, 2019 ), and open-set recognition ( Ren et al, 2021 ; Mandivarapu et al, 2022 ) gives a good overview over deep active learning approaches.…”