Active learning using uncertainty information

Yang, Yazhou; Loog, Marco

doi:10.1109/icpr.2016.7900034

Cited by 45 publications

(31 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rubens and Sugiyama [20] study influence-based collaborative active learning, where multiple users may work together to provide a result, such as a group-based recommendation. Yang and Loog [21] also study active learning using uncertain information. With regards to understanding deep neural network structures, Liu et al [22] look at understanding the training process of deep generative models, Ming et al [23] focus on understanding the hidden memories of recurrent neural networks, and Kahng et al [24] propose ActiVis as a visual analytics tool for exploring large deep neural networks.…”

Section: Related Workmentioning

confidence: 99%

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Legg

Smith

Downing

2019

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

Active machine learning is a human-centric paradigm that leverages a small labelled dataset to build an initial weak classifier, that can then be improved over time through human-machine collaboration. As new unlabelled samples are observed, the machine can either provide a prediction, or query a human 'oracle' when the machine is not confident in its prediction. Of course, just as the machine may lack confidence, the same can also be true of a human 'oracle': humans are not all-knowing, untiring oracles. A human's ability to provide an accurate and confident response will often vary between queries, according to the duration of the current interaction, their level of engagement with the system, and the difficulty of the labelling task. This poses an important question of how uncertainty can be expressed and accounted for in a human-machine collaboration. In short, how can we facilitate a mutually-transparent collaboration between two uncertain actors-a person and a machine-that leads to an improved outcome? In this work, we demonstrate the benefit of human-machine collaboration within the process of active learning, where limited data samples are available or where labelling costs are high. To achieve this, we developed a visual analytics tool for active learning that promotes transparency, inspection, understanding and trust, of the learning process through human-machine collaboration. Fundamental to the notion of confidence, both parties can report their level of confidence during active learning tasks using the tool, such that this can be used to inform learning. Human confidence of labels can be accounted for by the machine, the machine can query for samples based on confidence measures, and the machine can report confidence of current predictions to the human, to further the trust and transparency between the collaborative parties. In particular, we find that this can improve the robustness of the classifier when incorrect sample labels are provided, due to unconfidence or fatigue. Reported confidences can also better inform human-machine sample selection in collaborative sampling. Our experimentation compares the impact of different selection strategies for acquiring samples: machine-driven, human-driven, and collaborative selection. We demonstrate how a collaborative approach can improve trust in the model robustness, achieving high accuracy and low user correction, with only limited data sample selections.

show abstract

Section: Related Workmentioning

confidence: 99%

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Legg

Smith

Downing

2019

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…Uncertainty-based metrics are widely deployed in the AL field as the literature suggests [ 34 ], mainly due to their strong performance in terms of calculation efficiency and effectiveness in the process of selecting the most confusing instances. On the other hand, in the SSL field research works exist [ 8 , 35 ] proving the effectiveness of probabilistic iterative schemes.…”

Section: Proposed Methodsmentioning

confidence: 99%

Combination of Active Learning and Semi-Supervised Learning under a Self-Training Scheme

Fazakis

Kanas

Aridas

et al. 2019

Entropy

View full text Add to dashboard Cite

One of the major aspects affecting the performance of the classification algorithms is the amount of labeled data which is available during the training phase. It is widely accepted that the labeling procedure of vast amounts of data is both expensive and time-consuming since it requires the employment of human expertise. For a wide variety of scientific fields, unlabeled examples are easy to collect but hard to handle in a useful manner, thus improving the contained information for a subject dataset. In this context, a variety of learning methods have been studied in the literature aiming to efficiently utilize the vast amounts of unlabeled data during the learning process. The most common approaches tackle problems of this kind by individually applying active learning or semi-supervised learning methods. In this work, a combination of active learning and semi-supervised learning methods is proposed, under a common self-training scheme, in order to efficiently utilize the available unlabeled data. The effective and robust metrics of the entropy and the distribution of probabilities of the unlabeled set, to select the most sufficient unlabeled examples for the augmentation of the initial labeled set, are used. The superiority of the proposed scheme is validated by comparing it against the base approaches of supervised, semi-supervised, and active learning in the wide range of fifty-five benchmark datasets.

show abstract

“…The min-max view active learning directly measures the value of objective function during retraining procedure and selects the instance with minimum score in the worst case scenario. Recently, Yang & Loog (2016) proposed to improve the retraining-based algorithms by integrating the uncertainty information in the selection criterion.…”

Section: Related Workmentioning

confidence: 99%

A variance maximization criterion for active learning

Yang

Loog

2018

Pattern Recognition

Self Cite

View full text Add to dashboard Cite

Active learning aims to train a classifier as fast as possible with as few labels as possible. The core element in virtually any active learning strategy is the criterion that measures the usefulness of the unlabeled data based on which new points to be labeled are picked. We propose a novel approach which we refer to as maximizing variance for active learning or MVAL for short. MVAL measures the value of unlabeled instances by evaluating the rate of change of output variables caused by changes in the next sample to be queried and its potential labelling. In a sense, this criterion measures how unstable the classifier's output is for the unlabeled data points under perturbations of the training data. MVAL maintains, what we refer to as, retraining information matrices to keep track of these output scores and exploits two kinds of variance to measure the informativeness and representativeness, respectively. By fusing these variances, MVAL is able to select the instances which are both informative and representative. We employ our technique both in combination with logistic regression and support vector machines and demonstrate that MVAL achieves state-of-the-art performance in experiments on a large number of standard benchmark datasets.

show abstract

Active learning using uncertainty information

Cited by 45 publications

References 18 publications

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Combination of Active Learning and Semi-Supervised Learning under a Self-Training Scheme

A variance maximization criterion for active learning

Contact Info

Product

Resources

About