Leveraging User Paraphrasing Behavior In Dialog Systems To Automatically Collect Annotations For Long-Tail Utterances

Falke, Tobias; Boese, Markus; Sorokin, Daniil; Tırkaz, Çağlar; Lehnen, Patrick

doi:10.18653/v1/2020.coling-industry.3

Cited by 13 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MARUPA (Falke et al, 2020) (Mining Annotations from User Paraphrasing) is a tool to leverage realworld user implicit feedback to collect paraphrased utterances. Sometimes when a user is having a failed interaction with the system, the user will paraphrase the utterance and retry.…”

Section: Paraphrasing By User Feedbackmentioning

confidence: 99%

Efficient Semi-supervised Consistency Training for Natural Language Understanding

Leung¹,

Tan²

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Manually labeled training data is expensive, noisy, and often scarce, such as when developing new features or localizing existing features for a new region. In cases where labeled data is limited but unlabeled data is abundant, semi-supervised learning methods such as consistency training can be used to improve model performance, by training models to output consistent predictions between original and augmented versions of unlabeled data.In this work, we explore different data augmentation methods for consistency training (CT) on Natural Language Understanding (NLU) domain classification (DC) in the limited labeleddata regime. We explore three types of augmentation techniques (human paraphrasing, backtranslation, and dropout) for unlabeled data and train DC models to jointly minimize both the supervised loss and the consistency loss on unlabeled data. Our results demonstrate that DC models trained with CT methods and dropoutbased augmentation on only 0.1% (2,998 instances) of labeled data with the remainder as unlabeled can achieve a top-1 relative accuracy reduction of 12.25% compared to fully supervised model trained with 100% of labeled data, outperforming fully supervised models trained on 10x that amount of labeled data. The dropout-based augmentation achieves similar performance compare to back-translationbased augmentation with much less computational resources. This paves the way for applications of using large scale unlabeled data for semi-supervised learning in production NLU systems.

show abstract

Section: Paraphrasing By User Feedbackmentioning

confidence: 99%

Efficient Semi-supervised Consistency Training for Natural Language Understanding

Leung¹,

Tan²

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…Another challenge of building a skill recommender system is noisy and hard utterances that cannot be handled by NLU. Customers express their intents in many different ways with a long tail of rare utterances (Falke et al, 2020 ) which are hard for the voice assistant to interpret. Although these utterances are rare, in aggregate their volume is huge.…”

Section: Introductionmentioning

confidence: 99%

FROST: Fallback Voice Apps Recommendation for Unhandled Voice Commands in Intelligent Personal Assistants

Mohamed

Xiao

et al. 2022

Front. Big Data

View full text Add to dashboard Cite

Intelligent personal assistants (IPAs) such as Amazon Alexa, Google Assistant and Apple Siri extend their built-in capabilities by supporting voice apps developed by third-party developers. Sometimes the smart assistant is not able to successfully respond to user voice commands (aka utterances). There are many reasons including automatic speech recognition (ASR) error, natural language understanding (NLU) error, routing utterances to an irrelevant voice app, or simply that the user is asking for a capability that is not supported yet. The failure to handle a voice command leads to customer frustration. In this article, we introduce a fallback skill recommendation system (FROST) to suggest a voice app to a customer for an unhandled voice command. There are several practical issues when developing a skill recommender system for IPAs, i.e., partial observation, hard and noisy utterances. To solve the partial observation problem, we propose collaborative data relabeling (CDR) method. To mitigate hard and noisy utterance issues, we propose a rephrase-based relabeling technique. We evaluate the proposed system in both offline and online settings. The offline evaluation results show that the FROST system outperforms the baseline rule-based system. The online A/B testing results show a significant gain of customer experience metrics.

show abstract

“…Dialogue comprehension (Sun et al, 2019;Cui et al, 2020) aims to capture diverse kinds of key information in utterances, which are either scattered around or implicitly implied in different turns of conversations. Therefore, it requires different capabilities such as paraphrasing (Falke et al, 2020), summarizing (Gliwa et al, 2019), and commonsense reasoning (Arabshahi et al, 2021). Recent advances in pre-trained language models (PLMs) (Devlin et al, 2019;Radford et al, 2019) have been applied to the problem (Jin et al, 2020;Liu et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehension

Zhao¹,

Yao²,

Yu³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

Comprehending a dialogue requires a model to capture diverse kinds of key information in the utterances, which are either scattered around or implicitly implied in different turns of conversations. Therefore, dialogue comprehension requires diverse capabilities such as paraphrasing, summarizing, and commonsense reasoning. Towards the objective of pre-training a zero-shot dialogue comprehension model, we develop a novel narrative-guided pre-training strategy that learns by narrating the key information from a dialogue input. However, the dialoguenarrative parallel corpus for such a pre-training strategy is currently unavailable. For this reason, we first construct a dialogue-narrative parallel corpus by automatically aligning movie subtitles and their synopses. We then pre-train a BART model on the data and evaluate its performance on four dialogue-based tasks that require comprehension. Experimental results show that our model not only achieves superior zero-shot performance but also exhibits stronger finegrained dialogue comprehension capabilities. The data and code are available at https: //github.com/zhaochaocs/Diana.

show abstract

Leveraging User Paraphrasing Behavior In Dialog Systems To Automatically Collect Annotations For Long-Tail Utterances

Cited by 13 publications

References 21 publications

Efficient Semi-supervised Consistency Training for Natural Language Understanding

Efficient Semi-supervised Consistency Training for Natural Language Understanding

FROST: Fallback Voice Apps Recommendation for Unhandled Voice Commands in Intelligent Personal Assistants

Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehension

Contact Info

Product

Resources

About