SLiQA-I: Towards Cold-start Development of End-to-end Spoken Language Interface for Question Answering

Hsu

Ray

et al. 2021

Preprint

Self Cite

Intent classification is a major task in spoken language understanding (SLU). Since most models are built with pre-collected in-domain (IND) training utterances, their ability to detect unsupported out-of-domain (OOD) utterances has a critical effect in practical use. Recent works have shown that using extra data and labels can improve the OOD detection performance, yet it could be costly to collect such data. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection. Our method designs a novel domain-regularized module (DRM) to reduce the overconfident phenomenon of a vanilla classifier, achieving a better generalization in both cases. Besides, DRM can be used as a drop-in replacement for the last layer in any neural network-based intent classifier, providing a low-cost strategy for a significant improvement. The evaluation on four datasets shows that our method built on BERT and RoBERTa models achieves state-of-the-art performance against existing approaches and the strong baselines we created for the comparisons.

Section: Introductionmentioning

confidence: 99%

Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU

Hsu

Ray

et al. 2021

Preprint

Self Cite

“…Thus, such pretrained SLU models suffer from significant performance drop and fail to understand user utterances. Recently, [14,15] developed a cold start natural language generation algorithms to enrich the training data with the hope of covering more varieties with low cost. As an alternative approach, [16,17] proposed to leverage user and contextual information to mitigate the practical performance discrepancy.…”

Section: Introductionmentioning

confidence: 99%

Interpreting and Improving Deep Neural SLU Models via Vocabulary Importance

Chen²,

Jin

2019

Interspeech 2019

Self Cite

Spoken language understanding (SLU) is a crucial component in virtual personal assistants. It consists of two main tasks: intent detection and slot filling. State-of-the-art deep neural SLU models have demonstrated good performance on benchmark datasets. However, these models suffer from the significant performance drop in practice after deployment due to the data distribution discrepancy between training and real user utterances. In this paper, we first propose four research questions that help to understand what the state-of-the-art deep neural SLU models actually learn. To answer them, we study the vocabulary importance using a novel Embedding Sparse Structure Learning (SparseEmb) approach. It can be applied onto various existing deep SLU models to efficiently prune the useless words without any additional manual hyperparameter tuning. We evaluate SparseEmb on benchmark datasets using two existing SLU models and answer the proposed research questions. Then, we utilize SparseEmb to sanitize the training data based on the selected useless words as well as the model re-validation during training. Using both benchmark and our collected testing data, we show that our sanitized training data helps to significantly improve the SLU model performance. Both SparseEmb and training data sanitization approaches can be applied onto any deep learning based SLU models.

“…Existing industrial PA products completely rely on software developers to build new skills by manually developing NLU engine and implementing action fulfillment. On one hand, while recent work CRUISE and SliQA-I (Shen et al, 2019) have introduced automatic training utterance and question generation approaches with lightweight human workload, they still require the involvement of software developers for NLU development through IDE tools. Another line of research is to personalize NLU engine in existing skills (Azaria et al, 2016;Ray et al, 2018;Wang et al, 2018a), yet they cannot support building new skills.…”

Section: Introductionmentioning

confidence: 99%

Untitled

Proceedings of the 2019 Conference of the North

Ray

Jin

et al. 2019

Self Cite

We present SkillBot that takes the first step to enable end users to teach new skills in personal assistants (PA). Unlike existing PA products that need software developers to build new skills via IDE tools, an end user can use SkillBot to build new skills just by naturally demonstrating the task on device screen. SkillBot automatically develops a natural language understanding (NLU) engine and implements the action without the need of coding. On both benchmark and inhouse datasets, we validate the competitive performance of SkillBot automatically built NLU. We also observe that it only takes a few minutes for an end user to build a new skill using SkillBot.