The Power of Ensembles for Active Learning in Image Classification

Beluch, William; Genewein, Tim; Nürnberger, Andreas; Köhler, Jan M.

doi:10.1109/cvpr.2018.00976

Cited by 472 publications

(285 citation statements)

References 31 publications

Supporting

Mentioning

236

Contrasting

Order By: Relevance

“…We found this effect to be consistent over all parametrizations of the Bootstrap-and MCCD-based versions we evaluated. Our results match recent findings (Beluch et al, 2018), where ensemble-based uncertainty estimators were compared against Monte-Carlo Dropout based ones for the case of active learning in image classification. Results presented in that work also showed that ensembles performed better and led to more calibrated uncertainty estimates.…”

Section: Discussion and Future Worksupporting

confidence: 90%

Uncertainty-based Out-of-Distribution Classification in Deep Reinforcement Learning

Sedlmeier

Gabor

Phan

et al. 2020

Proceedings of the 12th International Conference on Agents and Artificial Intelligence

View full text Add to dashboard Cite

Robustness to out-of-distribution (OOD) data is an important goal in building reliable machine learning systems. Especially in autonomous systems, wrong predictions for OOD inputs can cause safety critical situations. As a first step towards a solution, we consider the problem of detecting such data in a value-based deep reinforcement learning (RL) setting. Modelling this problem as a one-class classification problem, we propose a framework for uncertainty-based OOD classification: UBOOD. It is based on the effect that an agent's epistemic uncertainty is reduced for situations encountered during training (in-distribution), and thus lower than for unencountered (OOD) situations. Being agnostic towards the approach used for estimating epistemic uncertainty, combinations with different uncertainty estimation methods, e.g. approximate Bayesian inference methods or ensembling techniques are possible. We further present a first viable solution for calculating a dynamic classification threshold, based on the uncertainty distribution of the training data. Evaluation shows that the framework produces reliable classification results when combined with ensemble-based estimators, while the combination with concrete dropout-based estimators fails to reliably detect OOD situations. In summary, UBOOD presents a viable approach for OOD classification in deep RL settings by leveraging the epistemic uncertainty of the agent's value function.

show abstract

Section: Discussion and Future Worksupporting

confidence: 90%

Uncertainty-based Out-of-Distribution Classification in Deep Reinforcement Learning

Sedlmeier

Gabor

Phan

et al. 2020

Proceedings of the 12th International Conference on Agents and Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…This method has been verified only with small-scale classification tasks. [4] constructs a committee comprising 5 deep networks to measure disagreement as uncertainty. It has shown the state-of-the-art classification performance, but it is also inefficient in terms of memory and computation for large-scale problems.…”

Section: Related Researchmentioning

confidence: 99%

“…Dataset We choose CIFAR-10 dataset [22] as it has been used for recent active learning methods [45,4]. CIFAR-10 consists of 60,000 images of 32×32×3 size, assigned with one of 10 object categories.…”

Section: Image Classificationmentioning

confidence: 99%

“…Given a pool of unlabeled data, there have been three major approaches according to the selection criteria: an uncertainty-based approach, a diversity-based approach, and expected model change. The uncertainty approach [26,19,55,52,49,4] defines and measures the quantity of uncertainty to select uncertain data points, while the diversity approach [45,37,15,5] selects diverse data points that represent the whole distribution of the unlabeled pool. Expected model change [44,48,12] selects data points that would cause the greatest change to the current model parameters or outputs if we knew their labels.…”

Section: Introductionmentioning

confidence: 99%

“…As a task-agnostic uncertainty approach, [49,4] train multiple models to construct a committee, and measure the consensus between the multiple predictions from the committee. However, constructing a committee is too expensive for current deep networks learned with large data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Loss for Active Learning

Yoo¹,

Kweon

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

557

520

View full text Add to dashboard Cite

The performance of deep neural networks improves with more annotated data. The problem is that the budget for annotation is limited. One solution to this is active learning, where a model asks human to annotate data that it perceived as uncertain. A variety of recent methods have been proposed to apply active learning to deep networks but most of them are either designed specific for their target tasks or computationally inefficient for large networks. In this paper, we propose a novel active learning method that is simple but task-agnostic, and works efficiently with the deep networks. We attach a small parametric module, named "loss prediction module," to a target network, and learn it to predict target losses of unlabeled inputs. Then, this module can suggest data that the target model is likely to produce a wrong prediction. This method is task-agnostic as networks are learned from a single loss regardless of target tasks. We rigorously validate our method through image classification, object detection, and human pose estimation, with the recent network architectures. The results demonstrate that our method consistently outperforms the previous methods over the tasks.

show abstract

How useful is active learning for image‐based plant phenotyping?

Nagasubramanian

Jubery

Ardakani

et al. 2021

The Plant Phenome Journal

View full text Add to dashboard Cite

Deep learning models have been successfully deployed for a diverse array of image-based plant phenotyping applications including disease detection and classification. However, successful deployment of supervised deep learning models requires large amount of labeled data, which is a significant challenge in plant sciences (and most biological) domain due to the inherent complexities. Specifically, data annotation is costly, laborious, time consuming and needs domain expertise for phenotyping tasks, especially for diseases. To overcome this challenge, active learning algorithms have been proposed to reduce the amount of labeling needed by deep learning models to achieve good predictive performance. Active learning methods work by adaptively suggesting samples to annotate using an acquisition function to achieve maximum (classification) performance under a fixed labeling budget. We report the performance of four different active learning methods, (1) Deep Bayesian Active Learning (DBAL), (2) Entropy, (3) Least Confidence, and (4) core-set, with conventional random sampling-based annotation for two vastly different image-based classification datasets. The first image dataset consists of soybean [Glycine max L. (Merr.)] leaves belonging to eight different soybean stresses and a healthy class, and the second consists of nine different weed species from the field. For a fixed labeling budget, we observed that the classification performance of deep learning models using active learning based acquisition strategies is better than random sampling-based acquisition for both datasets. The integration of active learning strategies for data annotation can help mitigate labelling challenges in the plant sciences applications particularly where resources dedicated to annotations are limited. INTRODUCTIONWith the advent of high throughput phenotyping in plant sciences (

show abstract

The Power of Ensembles for Active Learning in Image Classification

Cited by 472 publications

References 31 publications

Uncertainty-based Out-of-Distribution Classification in Deep Reinforcement Learning

Uncertainty-based Out-of-Distribution Classification in Deep Reinforcement Learning

Learning Loss for Active Learning

How useful is active learning for image‐based plant phenotyping?

Contact Info

Product

Resources

About