Cold-start Active Learning through Self-supervised Language Modeling

Yuan, Michelle; Lin, Hung‐Mo; Boyd-Graber, Jordan

doi:10.18653/v1/2020.emnlp-main.637

Cited by 90 publications

(93 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ruiter et al (2019) use an emergent NMT system to simultaneously select training data and learn internal NMT representations in a SSL way without parallel data. SSL is also adopted to solve many other problems, such as document-level context or sentence summarization (West et al, 2019;Wang et al, 2019b), dialogue learning (Wu et al, 2019), improving data scarcity or labeling costs (Fu et al, 2020;Yuan et al, 2020) and generating meta-learning tasks from unlabeled text (Bansal et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Curriculum Learning for Spelling Error Correction

Gan¹,

Xu²,

Hongying³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Spelling Error Correction (SEC) that requires high-level language understanding is a challenging but useful task. Current SEC approaches normally leverage a pre-training then fine-tuning procedure that treats data equally. By contrast, Curriculum Learning (CL) utilizes training data differently during training and has shown its effectiveness in improving both performance and training efficiency in many other NLP tasks. In NMT, a model's performance has been shown sensitive to the difficulty of training examples, and CL has been shown effective to address this. In SEC, the data from different language learners are naturally distributed at different difficulty levels (some errors made by beginners are obvious to correct while some made by fluent speakers are hard), and we expect that designing a curriculum correspondingly for model learning may also help its training and bring about better performance. In this paper, we study how to further improve the performance of the state-of-the-art SEC method with CL, and propose a Self-Supervised Curriculum Learning (SSCL) approach. Specifically, we directly use the cross-entropy loss as criteria for: 1) scoring the difficulty of training data, and 2) evaluating the competence of the model. In our approach, CL improves the model training, which in return improves the CL measurement. In our experiments on the SIGHAN 2015 Chinese spelling check task, we show that SSCL is superior to previous norm-based and uncertainty-aware approaches, and establish a new state of the art (74.38% F1).

show abstract

Section: Related Workmentioning

confidence: 99%

Self-Supervised Curriculum Learning for Spelling Error Correction

Gan¹,

Xu²,

Hongying³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Ideally, active learning could be most useful in low-resource settings. In practice, however, it is more likely that the model might work poorly with the limited number of labelled data at the beginning of active learning [34]. Therefore, introducing a component to ensures a certain level of performance with the limited labelled data is important to address the cold-start issue.…”

Section: Related Work a Aspect-based Sentiment Analysismentioning

confidence: 99%

LETS: A Label-Efficient Training Scheme for Aspect-Based Sentiment Analysis by Using a Pre-Trained Language Model

Shim

Lowet²,

Luca

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Recently proposed pre-trained language models can be easily fine-tuned to a wide range of downstream tasks. However, a large-scale labelled task-specific dataset is required for fine-tuning creating a bottleneck in the development process of machine learning applications. To foster a fast development by reducing manual labelling efforts, we propose a Label-Efficient Training Scheme (LETS). The proposed LETS consists of three elements: (i) task-specific pre-training to exploit unlabelled task-specific corpus data, (ii) label augmentation to maximise the utility of labelled data, and (iii) active learning to label data strategically. In this paper, we apply LETS to a novel aspect-based sentiment analysis (ABSA) use-case for analysing the reviews of the health-related program supporting people to improve their sleep quality. We validate the proposed LETS on a custom health-related program-reviews dataset and another ABSA benchmark dataset. Experimental results show that the LETS can reduce manual labelling efforts 2-3 times compared to labelling with random sampling on both datasets. The LETS also outperforms other state-of-the-art active learning methods. Furthermore, the experimental results show that LETS can contribute to better generalisability with both datasets compared to other methods thanks to the task-specific pre-training and the proposed label augmentation. We expect this work could contribute to the natural language processing (NLP) domain by addressing the issue of the high cost of manually labelling data. Also, our work could contribute to the healthcare domain by introducing a new potential application of NLP techniques.

show abstract

“…While general purpose transformer-based language models are often used, fine-tuning them often require numerous examples (Dodge et al, 2020). One possible solution to address this lack of labelled data is to use active learning (Yuan et al, 2020) in conjunction with pre-trained models such as BERT (Devlin et al, 2019) that comes with possible bias problems (Papakyriakopoulos et al, 2020). In Cultural Heritage domains, several works are trying to compensate for the lack of labelled data.…”

Section: Related Workmentioning

confidence: 99%

Zero-Shot Information Extraction to Enhance a Knowledge Graph Describing Silk Textiles

Schleider¹,

Troncy²

2021

Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities An

View full text Add to dashboard Cite

The knowledge of the European silk textile production is a typical case for which the information collected is heterogeneous, spread across many museums and sparse since rarely complete. Knowledge Graphs for this cultural heritage domain, when being developed with appropriate ontologies and vocabularies, enable to integrate and reconcile this diverse information. However, many of these original museum records still have some metadata gaps. In this paper, we present a zero-shot learning approach that leverages the Concept-Net common sense knowledge graph to predict categorical metadata informing about the silk objects production. We compared the performance of our approach with traditional supervised deep learning-based methods that do require training data. We demonstrate promising and competitive performance for similar datasets and circumstances and the ability to predict sometimes more fine-grained information. Our results can be reproduced using the code and datasets published at https:// github.com/silknow/ZSL-KG-silk.

show abstract

Cold-start Active Learning through Self-supervised Language Modeling

Cited by 90 publications

References 39 publications

Self-Supervised Curriculum Learning for Spelling Error Correction

Self-Supervised Curriculum Learning for Spelling Error Correction

LETS: A Label-Efficient Training Scheme for Aspect-Based Sentiment Analysis by Using a Pre-Trained Language Model

Zero-Shot Information Extraction to Enhance a Knowledge Graph Describing Silk Textiles

Contact Info

Product

Resources

About