Data-Efficient Paraphrase Generation to Bootstrap Intent Classification and Slot Labeling for New Features in Task-Oriented Dialog Systems

Abstract

Recent progress through advanced neural models pushed the performance of task-oriented dialog systems to almost perfect accuracy on existing benchmark datasets for intent classification and slot labeling. However, in evolving real-world dialog systems, where new functionality is regularly added, a major additional challenge is the lack of annotated training data for such new functionality, as the necessary data collection efforts are laborious and time-consuming. A potential solution to reduce the effort is to augment initial seed data by paraphrasing existing utterances automatically. In this paper, we propose a new, data-efficient approach following this idea. Using an interpretation-to-text model for paraphrase generation, we are able to rely on existing dialog system training data, and, in combination with shuffling-based sampling techniques, we can obtain diverse and novel paraphrases from small amounts of seed data. In experiments on a public dataset and with a real-world dialog system, we observe improvements for both intent classification and slot labeling, demonstrating the usefulness of our approach.

Multilingual Paraphrase Generation For Bootstrapping New Features in Task-Oriented Dialog Systems

“…Our approach is focused towards generating utterances in the dialog domain that can generate utterances from a sequence of slots conditioned on both intent and language. Jolly et al (2020) showed that an interpretationto-text model can be used with shuffling-based sampling techniques to generate diverse and novel paraphrases from small amounts of seed data, that improve accuracy when augmenting to the existing training data. Our approach is different as our model can generate the slot annotations along with the the utterance, which are necessary for the slot labeling task.…”

“…However, labeled examples for the new feature are typically limited to a small set of seed examples, as the collection of more annotations would make feature expansion costly and slow. As a possible solution, previous work explored the automatic generation of paraphrases to augment the seed data (Malandrakis et al, 2019;Cho et al, 2019;Jolly et al, 2020).…”

“…To address this setup, we follow the recent work of Jolly et al (2020), which proposes to use an encoder-decoder model that maps from structured meaning representations to corresponding utterances. Because such an input is language-agnostic, it is particularly well-suited for the multilingual setup.…”

Identifying and Resolving Annotation Changes for Natural Language Understanding

“…Intent classification and slot labeling are two fundamental tasks in spoken language understanding, dating back to early 90's (Price, 1990). With the rise of task-oriented personal assistants, the two tasks got more attention and progress has been made by applying various deep learning techniques (Abujabal and Gaspers, 2019;Goo et al, 2018; Jolly et al, 2020;Mesnil et al, 2013;Zhang and Wang, 2016). While we focus on resolving annotation conflicts for NLU with linear labeling i.e., intent and slot labels, our approach can be still used for other more complex tree-based labeling e.g., labeling dependency parses or ontology trees (Chen and Manning, 2014), with the minor change of replacing the task-specific neural LSTM-based classification model.…”

“…PlayMusic), and (2) A slot labeling (SL) model, which classifies tokens into slot types, out of a predefined set (e.g. SongName) (Goo et al, 2018;Jolly et al, 2020). An example utterance is shown in Figure 1, with two conflicting annotations.…”