The Use of Unlabeled Data Versus Labeled Data for Stopping Active Learning for Text Classification

Beatty, Garrett; Kochis, Ethan; Bloodgood, Michael

doi:10.1109/icosc.2019.8665546

Cited by 15 publications

(22 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of stopping criterion for active learning have been proposed, [77][78][79][80][81][82] but in this work we monitor and define convergence using the stabilizing predictions (SP) method that evaluates performance based on unlabeled data 81 and the performance difference (PD) method that considers the labeled examples. 83 The SP method examines the predictions of consecutive models at each iteration of the active learning procedure on a randomly selected set of 500 points, called the stop set, which is held constant throughout the active learning. We measure the difference in the regression predictions between subsequent rounds using the average Bhattacharyya distance D B 84 between the posterior of consecutive GPR models over the stop set.…”

Section: Stop Criteriamentioning

confidence: 99%

Discovery of Self-Assembling π-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation

et al. 2020

View full text Add to dashboard Cite

Electronically-active organic molecules have demonstrated great promise as novel soft materials for energy harvesting and transport. Self-assembled nanoaggregates formed from π-conjugated oligopeptides composed of an aromatic core flanked by oligopeptide wings offer emergent optoelectronic properties within a water soluble and biocompatible substrate. Nanoaggregate properties can be controlled by tuning core chemistry and peptide composition, but the sequence-structure-function relations remain poorly characterized. In this work, we employ coarse-grained molecular dynamics simulations within an active learning protocol employing deep representational learning and Bayesian optimization to efficiently identify molecules capable of assembling pseudo-1D nanoaggregates with good stacking of the electronically-active π-cores. We consider the DXXX-OPV3-XXXD oligopeptide family, where D is an Asp residue and OPV3 is an oligophenylene vinylene oligomer (1,4-distyrylbenzene), to identify the top performing XXX tripeptides within all 20 3 = 8,000 possible sequences. By direct simulation of only 2.3% of this space, we identify molecules predicted to exhibit superior assembly relative to those reported in prior work. Spectral clustering of the top candidates reveals new design rules governing assembly. This work establishes new understanding of DXXX-OPV3-XXXD assembly, identifies promising new candidates for experimental testing, and presents a computational design platform that can be generically extended to other peptide-based and peptide-like systems.

show abstract

Section: Stop Criteriamentioning

confidence: 99%

Discovery of Self-Assembling π-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…On the other hand, an essential aspect to consider in AL is determining a criterion for stopping the learning process. Some of the stopping criteria analyze the cost of obtaining new labels, set a maximum performance value for the classifier or training sample size, or analyze the quality of the examples in the datasets [12], [31]- [35]. One approach to a stopping criteria method was proposed by Bloodgood and Vijay-Shanker considering the unlabeled dataset [34] The method tests the new models obtained in consecutive iterations of AL in a separate dataset without labels to check if the predictions have stabilized.…”

Section: Related Workmentioning

confidence: 99%

Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions

2021

View full text Add to dashboard Cite

Biomedical text classification algorithms, which currently support clinical decision-making processes, call for expensive training texts due to the low availability of labeled corpus and the cost of manual annotation by specialized professionals. The active learning (AL) approach to classification heavily lessens such cost by reducing the number of labeled documents required to achieve specified performance. This article introduces a query strategy and a stopping criterion that transform CREGEX, a regular-expressions-based text classification algorithm, in an AL biomedical text classifier. The query strategy samples the training dataset, trading off the greedy learning achieved by the regular expressions classification precision and the conservative learning induced by text sequence alignment classification. The sustained reduction in the variance of the query strategy scores is used as a stopping criterion. The AL classifier was compared with Support Vector Machine (SVM), Naïve Bayes (NB), and a classifier based on Bidirectional Encoder Representations from Transformers (BERT), using three datasets with biomedical information in Spanish on smoking habits, obesity, and obesity types. The learning curve results indicate that AL in CREGEX allowed to efficiently reduce the number of training examples for equal performance than the rest of the classifiers, obtaining areas under the learning curve greater than 85% in all cases. The stopping criterion applied to the AL process allowed to use, on average, approximately 32% to 50% of the total training examples with differences in performance concerning the maximum value of the learning curve not exceeding 2%. This performance demonstrates the effectiveness of using AL in a biomedical text classifier based on regular expressions, which is attributable to such expressions' ability to represent intricate sequential patterns in training texts considered most informative.

show abstract

“…Also, to decide on the stopping criteria for AL, we examined the learning curves and stopped the process when the classifier performance shows no improvement with additional iterations [17]. We use λ= 0.0001 as a threshold of performance differences and stop the experiments when the mean of performance differences does not exceed λ for a successive number of iterations.…”

Section: Experiments Settingsmentioning

confidence: 99%

WeSAL: Applying Active Supervision to Find High-quality Labels at Industrial Scale

Nashaat¹,

Ghosh²,

Miller³

et al. 2020

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

Obtaining hand-labeled training data is one of the most tedious and expensive parts of the machine learning pipeline. Previous approaches, such as active learning aim at optimizing user engagement to acquire accurate labels. Other methods utilize weak supervision to generate low-quality labels at scale. In this paper, we propose a new hybrid method named WeSAL that incorporates Weak Supervision sources with Active Learning to keep humans in the loop. The method aims to generate large-scale training labels while enhancing its quality by involving domain experience. To evaluate WeSAL, we compare it with two-state-of-the-art labeling techniques, Active Learning, and Data Programming. The experiments use five publicly available datasets and a real-world dataset of 1.5M records provided by our industrial partner, IBM. The results indicate that WeSAL can generate large-scale, high-quality labels while reducing the labeling cost by up to 68% compared to active learning.

show abstract

The Use of Unlabeled Data Versus Labeled Data for Stopping Active Learning for Text Classification

Cited by 15 publications

References 25 publications

Discovery of Self-Assembling π-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation

Discovery of Self-Assembling π-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation

Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions

WeSAL: Applying Active Supervision to Find High-quality Labels at Industrial Scale

Contact Info

Product

Resources

About