PT4AL: Using Self-Supervised Pretext Tasks for Active Learning

Yi, John Seon Keun; Seo, Minseok; Park, Jongchan; Choi, Dong‐Geol

doi:10.48550/arxiv.2201.07459

Cited by 3 publications

(37 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Like classical active learning, deep active learning uses a variety of criteria for selecting data points. Uncertainty sampling is a major direction here as well [Gal et al, 2017, Yi et al, 2022, Ren et al, 2021, Zhan et al, 2022. A well-known state-of-the-art method in this context is MC Dropout [Gal et al, 2017], which uses dropout at test time in order to create variants of an initially trained deep neural network model; averaging predictions over these model variants then yields estimates of class probabilities that can be used by standard acquisition functions.…”

Section: Related Workmentioning

confidence: 99%

“…Some methods combine self-supervised or semisupervised methods with standard active learning methods. For instance, the method by [Yi et al, 2022] performs a pretext task (like rotation prediction) over the pool of unlabeled data points, which is then divided into batches based on the performance on the pretext task. Then, uncertainty-based active sampling is performed over one batch.…”

Section: Related Workmentioning

confidence: 99%

“…We performed two tests. The first one is similar to the setups used by Siméoni et al [2021] and Yi et al [2022], with 1K initial training data points and nine acquisitions in batches of size 1K. The second one (performed only for Max Entropy) used 5K initial training data points and four acquisition batches of 5K points each.…”

Section: Experimental Analysismentioning

confidence: 99%

“…Uncertainty-based approaches consider a point informative if its class label is highly uncertain given the current training data [Lewis and Gale, 1994]. By contrast, distribution-based methods intuitively assume that points are informative if they lie in a relatively dense area (according to some underlying data distribution) from which not many training points have been sampled yet [Yi et al, 2022].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Actively Learning Deep Neural Networks with Uncertainty Sampling Based on Sum-Product Networks

Khosravani¹,

Zilles²

2022

Preprint

View full text Add to dashboard Cite

Active learning is a popular approach for reducing the amount of data in training deep neural network models. Its success hinges on the choice of an effective acquisition function, which ranks not yet labeled data points according to their expected informativeness. In uncertainty sampling, the uncertainty that the current model has about a point's class label is the main criterion for this type of ranking. This paper proposes a new approach to uncertainty sampling in training a Convolutional Neural Network (CNN). The main idea is to use the feature representation extracted by the CNN as data for training a Sum-Product Network (SPN). Since SPNs are typically used for estimating the distribution of a dataset, they are well suited to the task of estimating class probabilities that can be used directly by standard acquisition functions such as max entropy and variational ratio. Moreover, we enhance these acquisition functions by weights calculated with the help of the SPN model; these weights make the acquisition function more sensitive to the diversity of conceivable class labels for a data point. The effectiveness of our method is demonstrated in an experimental study on the MNIST, Fashion-MNIST and CIFAR-10 datasets, where we compare it to the state-of-the-art methods MC Dropout and Bayesian Batch.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Experimental Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Actively Learning Deep Neural Networks with Uncertainty Sampling Based on Sum-Product Networks

Khosravani¹,

Zilles²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Another critical aspect is balancing uncertainty and diversity to label a more reasonable distribution of samples. Existing methods, such as PT4AL [18], employ equidistant disjoint sub-pools, but determining the size of each sub-pool poses challenges. A small sub-pool has a greater impact on the order of CDDs, limiting the choice of the uncertainty-based sampler.…”

Section: Introductionmentioning

confidence: 99%

Diversity-Boosted Generalization-Specialization Balancing for Zero-Shot Learning

Liu

Chang

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

The objective of Active Learning is to strategically label a subset of the dataset to maximize performance within a predetermined labeling budget. In this study, we harness features acquired through self-supervised learning. We introduce a straightforward yet potent metric, Cluster Distance Difference, to identify diverse data. Subsequently, we introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data. Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%. Moreover, we assess the efficacy of our proposed framework under extended settings, encompassing both larger and smaller labeling budgets. Experimental results demonstrate that, when labeling 80% of the samples, the performance of the current SOTA method declines by 0.74%, whereas our proposed BAL achieves performance comparable to the full dataset. Codes are available at https://github.com/JulietLJY/BAL.

show abstract

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

Kinakh,

Drozdova,

Voloshynovskiy

2024

Entropy

View full text Add to dashboard Cite

We present a new method of self-supervised learning and knowledge distillation based on multi-views and multi-representations (MV–MR). MV–MR is based on the maximization of dependence between learnable embeddings from augmented and non-augmented views, jointly with the maximization of dependence between learnable embeddings from the augmented view and multiple non-learnable representations from the non-augmented view. We show that the proposed method can be used for efficient self-supervised classification and model-agnostic knowledge distillation. Unlike other self-supervised techniques, our approach does not use any contrastive learning, clustering, or stop gradients. MV–MR is a generic framework allowing the incorporation of constraints on the learnable embeddings via the usage of image multi-representations as regularizers. The proposed method is used for knowledge distillation. MV–MR provides state-of-the-art self-supervised performance on the STL10 and CIFAR20 datasets in a linear evaluation setup. We show that a low-complexity ResNet50 model pretrained using proposed knowledge distillation based on the CLIP ViT model achieves state-of-the-art performance on STL10 and CIFAR100 datasets.

show abstract

PT4AL: Using Self-Supervised Pretext Tasks for Active Learning

Cited by 3 publications

References 0 publications

Actively Learning Deep Neural Networks with Uncertainty Sampling Based on Sum-Product Networks

Actively Learning Deep Neural Networks with Uncertainty Sampling Based on Sum-Product Networks

Diversity-Boosted Generalization-Specialization Balancing for Zero-Shot Learning

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

Contact Info

Product

Resources

About