On the Robustness of Active Learning

Hahn, Lukas; Roese-Koerner, Lutz; Cremer, Peet; Zimmermann, Urs; Maoz, Ori; Kummert, Anton

doi:10.29007/thws

Cited by 2 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While two of three baselines, i.e., MC-Dropout and Ensemble, beat the random acquisition, CoreSet does not reach a comparable classification accuracy within the labeling budget. This finding is in line with previous research (Sinha et al, 2019;Hahn et al, 2019a) and can be attributed to a weakness of the utilized p-norm distance metric regarding high-dimensional data, called the distance concentration phenomenon.…”

Section: Policy Trainingsupporting

confidence: 92%

“…Our method consistently outperforms the other approaches on both datasets. On FMNIST our method is the only method that is actually able to beat a Random Sampling strategy (similar findings have previously been reported by Hahn et al (2019b)). While our method is consistently 1 − 3% better than the Random Sampling strategy on FMNIST, on the harder KMNIST dataset we are 7.9% ahead on average.…”

Section: Policy Transfersupporting

confidence: 86%

See 1 more Smart Citation

IALE: Imitating Active Learner Ensembles

Loeffler¹,

Mutschler²

2020

Preprint

View full text Add to dashboard Cite

Active learning (AL) prioritizes the labeling of the most informative data samples. As the performance of well-known AL heuristics highly depends on the underlying model and data, recent heuristic-independent approaches that are based on reinforcement learning directly learn a policy that makes use of the labeling history to select the next sample. However, those methods typically need a huge number of samples to sufficiently explore the relevant state space. Imitation learning approaches aim to help out but again rely on a given heuristic. This paper proposes an improved imitation learning scheme that learns a policy for batch-mode pool-based AL. This is similar to previously presented multi-armed bandit approaches but in contrast to them we train a policy that imitates the selection of the best expert heuristic at each stage of the AL cycle directly. We use DAGGER to train the policy on a dataset and later apply it to similar datasets. With multiple AL heuristics as experts, the policy is able to reflect the choices of the best AL heuristics given the current state of the active learning process. We evaluate our method on well-known image datasets and show that we outperform state of the art imitation learners and heuristics.

show abstract

Section: Policy Trainingsupporting

confidence: 92%

Section: Policy Transfersupporting

confidence: 86%

IALE: Imitating Active Learner Ensembles

Loeffler¹,

Mutschler²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…At the heart of an AL method is the so-called query strategy that decides which data to present to the annotator / oracle next. In general, uncertainty sampling is one of the most common query strategies [10,38,1,13,29], besides that there also exist approaches on expected model change [36] and reinforcement learning based AL [2].…”

Section: Introductionmentioning

confidence: 99%

MetaBox+: A new Region Based Active Learning Method for Semantic Segmentation using Priority Maps

Colling¹,

Roese-Koerner²,

Gottschalk³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We present a novel region based active learning method for semantic image segmentation, called MetaBox+. For acquisition, we train a meta regression model to estimate the segment-wise Intersection over Union (IoU ) of each predicted segment of unlabeled images. This can be understood as an estimation of segment-wise prediction quality. Queried regions are supposed to minimize to competing targets, i.e., low predicted IoU values / segmentation quality and low estimated annotation costs. For estimating the latter we propose a simple but practical method for annotation cost estimation. We compare our method to entropy based methods, where we consider the entropy as uncertainty of the prediction. The comparison and analysis of the results provide insights into annotation costs as well as robustness and variance of the methods. Numerical experiments conducted with two different networks on the Cityscapes dataset clearly demonstrate a reduction of annotation effort compared to random acquisition. Noteworthily, we achieve 95% of the mean Intersection over Union (mIoU ), using MetaBox+ compared to when training with the full dataset, with only 10.47% / 32.01% annotation effort for the two networks, respectively.

show abstract

On the Robustness of Active Learning

Cited by 2 publications

References 14 publications

IALE: Imitating Active Learner Ensembles

IALE: Imitating Active Learner Ensembles

MetaBox+: A new Region Based Active Learning Method for Semantic Segmentation using Priority Maps

Contact Info

Product

Resources

About