Stopping Active Learning Based on Predicted Change of F Measure for Text Classification

Altschuler, Michael; Bloodgood, Michael

doi:10.1109/icosc.2019.8665646

Cited by 16 publications

(21 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section describes the evaluation of the proposed active learning method via a set of multi-class classification experiments with both artificial and real-world datasets 2 . We first investigate which dataset is useful for learning the acquisition function.…”

Section: Methodsmentioning

confidence: 99%

“…In such cases, we encounter another problem, namely, when to stop learning. There are several works on the optimal stopping timing of active learning [2,4,22,26]. There are only few works in the literature of active learning in which the budget is explicitly considered [7,13], where the authors derived a budget aware stream-based active learning, which do not consider learning the acquisition function from data.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Pre-Training Acquisition Functions by Deep Reinforcement Learning for Fixed Budget Active Learning

Taguchi

Hino

Kameyama

2021

Neural Process Lett

View full text Add to dashboard Cite

There are many situations in supervised learning where the acquisition of data is very expensive and sometimes determined by a user’s budget. One way to address this limitation is active learning. In this study, we focus on a fixed budget regime and propose a novel active learning algorithm for the pool-based active learning problem. The proposed method performs active learning with a pre-trained acquisition function so that the maximum performance can be achieved when the number of data that can be acquired is fixed. To implement this active learning algorithm, the proposed method uses reinforcement learning based on deep neural networks as as a pre-trained acquisition function tailored for the fixed budget situation. By using the pre-trained deep Q-learning-based acquisition function, we can realize the active learner which selects a sample for annotation from the pool of unlabeled samples taking the fixed-budget situation into account. The proposed method is experimentally shown to be comparable with or superior to existing active learning methods, suggesting the effectiveness of the proposed approach for the fixed-budget active learning.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Pre-Training Acquisition Functions by Deep Reinforcement Learning for Fixed Budget Active Learning

Taguchi

Hino

Kameyama

2021

Neural Process Lett

View full text Add to dashboard Cite

show abstract

“…However, our models can then forecast levels of performance in addition to only changes. Future work includes developing algorithms to combine the mathematical bounds approach of [15], [21] with the regression approach in the current paper.…”

Section: Related Workmentioning

confidence: 99%

“…This shows the need for improving the state-of-the-art so that we can forecast effectively in active learning settings. Future work that could be promising for accomplishing this includes developing algorithms to combine the mathematical bounds approach of [15], [21] with the regression approach in the current paper.…”

Section: E Impact Of Passive Learning Vs Active Learningmentioning

confidence: 99%

See 1 more Smart Citation

Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning

Orth

Bloodgood

2020

2020 IEEE 14th International Conference on Semantic Computing (ICSC)

Self Cite

View full text Add to dashboard Cite

When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stopping methods to minimize the cost of data annotation. An important capability for improving the utility of stopping methods is to effectively forecast the performance of the text classification models. Forecasting can be done through the use of logarithmic models regressed on some portion of the data as learning is progressing. A critical unexplored question is what portion of the data is needed for accurate forecasting. There is a tension, where it is desirable to use less data so that the forecast can be made earlier, which is more useful, versus it being desirable to use more data, so that the forecast can be more accurate. We find that when using active learning it is even more important to generate forecasts earlier so as to make them more useful and not waste annotation effort. We investigate the difference in forecasting difficulty when using accuracy and F-measure as the text classification system performance metrics and we find that F-measure is more difficult to forecast. We conduct experiments on seven text classification datasets in different semantic domains with different characteristics and with three different base machine learning algorithms. We find that forecasting is easiest for decision tree learning, moderate for Support Vector Machines, and most difficult for neural networks.

show abstract

IFHS Method on Moving Object Detection in Vehicle Flow

Zhao

Wei

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

Stopping Active Learning Based on Predicted Change of F Measure for Text Classification

Cited by 16 publications

References 23 publications

Pre-Training Acquisition Functions by Deep Reinforcement Learning for Fixed Budget Active Learning

Pre-Training Acquisition Functions by Deep Reinforcement Learning for Fixed Budget Active Learning

Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning

IFHS Method on Moving Object Detection in Vehicle Flow

Contact Info

Product

Resources

About