Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity

Cho, Eunah; Xie, He; Lalor, John P.; Kumar, Varun; Campbell, William M.

doi:10.1109/asru46091.2019.9003747

Cited by 15 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Selection by Submodular Optimization: Submodular data selection is used to select a diverse representative subset of samples from given dataset. This method has been applied in speech recognition (Wei et al, 2015), machine translation (Kirchhoff and Bilmes, 2014) and natural language understanding tasks (Cho et al, 2019). For SSL data selection, we use feature-based submodular selection (Kirchhoff and Bilmes, 2014), where submodular functions are given by weighted sums of nondecreasing concave functions applied to modular functions.…”

Section: Data Selection Approachesmentioning

confidence: 99%

Industry Scale Semi-Supervised Learning for Natural Language Understanding

Chen¹,

Garcia²,

Kumar³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

This paper presents a production Semi-Supervised Learning (SSL) pipeline based on the student-teacher framework, which leverages millions of unlabeled examples to improve Natural Language Understanding (NLU) tasks. We investigate two questions related to the use of unlabeled data in the production SSL context: 1) how to select samples from a huge unlabeled data pool that are beneficial for SSL training, and 2) how do the selected data affect the performance of different state-of-the-art SSL techniques. We compare four widely used SSL techniques, Pseudo-Label (PL), Knowledge Distillation (KD), Virtual Adversarial Training (VAT), and Cross-View Training (CVT) in conjunction with two data selection methods including committeebased selection and submodular optimizationbased selection. We further examine the benefits and drawbacks of these techniques when applied to intent classification (IC) and named entity recognition (NER) tasks in the English language using a public dataset (SNIPS) and real-world data from Amazon Alexa. To conclude we provide guidelines specifying when each of these methods might be beneficial to improve large-scale NLU systems.

show abstract

Section: Data Selection Approachesmentioning

confidence: 99%

Industry Scale Semi-Supervised Learning for Natural Language Understanding

Chen¹,

Garcia²,

Kumar³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

show abstract

“…Selection by Submodular Optimization: Sub-modular data selection is used to select a diverse representative subset of samples from given dataset. This method has been applied in speech recognition (Wei et al, 2015), machine translation (Kirchhoff and Bilmes, 2014) and natural language understanding tasks (Cho et al, 2019). For SSL data selection, we use feature-based submodular selection (Kirchhoff and Bilmes, 2014), where submodular functions are given by weighted sums of nondecreasing concave functions applied to modular functions.…”

Section: Data Selection Approachesmentioning

confidence: 99%

Industry Scale Semi-Supervised Learning for Natural Language Understanding

Chen

Garcia

Kumar

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a production Semi-Supervised Learning (SSL) pipeline based on the student-teacher framework, which leverages millions of unlabeled examples to improve Natural Language Understanding (NLU) tasks. We investigate two questions related to the use of unlabeled data in production SSL context: 1) how to select samples from a huge unlabeled data pool that are beneficial for SSL training, and 2) how do the selected data affect the performance of different state-of-the-art SSL techniques. We compare four widely used SSL techniques, Pseudo-Label (PL), Knowledge Distillation (KD), Virtual Adversarial Training (VAT) and Cross-View Training (CVT) in conjunction with two data selection methods including committeebased selection and submodular optimization based selection. We further examine the benefits and drawbacks of these techniques when applied to intent classification (IC) and named entity recognition (NER) tasks, and provide guidelines specifying when each of these methods might be beneficial to improve large scale NLU systems.

show abstract

“…We used two 100-dimension hidden layers with ReLU activation (Nair and Hinton, 2010) for the task. Further details of the embedding learning and classification model can be found in Cho et al (2019b).…”

Section: Paraphrase Classificationmentioning

confidence: 99%

Graph-Based Semi-Supervised Learning for Natural Language Understanding

Qiu¹,

Cho

Ma³

et al. 2019

Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13)

Self Cite

View full text Add to dashboard Cite

Semi-supervised learning is an efficient method to augment training data automatically from unlabeled data. Development of many natural language understanding (NLU) applications has a challenge where unlabeled data is relatively abundant while labeled data is rather limited. In this work, we propose transductive graphbased semi-supervised learning models as well as their inductive variants for NLU. We evaluate the approach's applicability using publicly available NLU data and models. In order to find similar utterances and construct a graph, we use a paraphrase detection model. Results show that applying the inductive graph-based semi-supervised learning can improve the error rate of the NLU model by 5%.

show abstract

Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity

Cited by 15 publications

References 27 publications

Industry Scale Semi-Supervised Learning for Natural Language Understanding

Industry Scale Semi-Supervised Learning for Natural Language Understanding

Industry Scale Semi-Supervised Learning for Natural Language Understanding

Graph-Based Semi-Supervised Learning for Natural Language Understanding

Contact Info

Product

Resources

About