A Progressive Model to Enable Continual Learning for Semantic Slot Filling

Abstract

Semantic slot filling is one of the major tasks in spoken language understanding (SLU). After a slot filling model is trained on precollected data, it is crucial to continually improve the model after deployment to learn users' new expressions. As the data amount grows, it becomes infeasible to either store such huge data and repeatedly retrain the model on all data or fine tune the model only on new data without forgetting old expressions. In this paper, we introduce a novel progressive slot filling model, ProgModel. ProgModel consists of a novel context gate that transfers previously learned knowledge to a small size expanded component; and meanwhile enables this new component to be fast trained to learn from new data. As such, ProgModel learns the new knowledge by only using new data at each time and meanwhile preserves the previously learned expressions. Our experiments show that ProgModel needs much less training time and smaller model size to outperform various model fine tuning competitors by up to 4.24% and 3.03% on two benchmark datasets.

Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation Networks

“…Lifelong learning, also called continual learning, is a long-standing research topic in machine learning, which enables models to perform online learning on new data (Cauwenberghs and Poggio, 2000;Kuzborskij et al, 2013). Architecture-based methods dynamically extend the model architecture to learn new data (Fernando et al, 2017;Shen et al, 2019). However, the model size grows rapidly with the increase of new data, which limits the application of architecture-based methods.…”

“…However, their setting is only a one-step incremental process. Shen et al (2019) continually train a slot-filling model on new data from the same domain. Madotto et al (2020) introduce continual learning into multiple dialogue tasks.…”

Improving Slot Filling by Utilizing Contextual Information

“…Also, joint models to simultaneously predict the intent of the utterance and to extract the semantic slots has also gained a lot of attention (Guo et al, 2014;Liu and Lane, 2016;Zhang and Wang, 2016;Wang et al, 2018;Goo et al, 2018;Qin et al, 2019;E et al, 2019). In addition to the supervised settings, recently other setting such as progressive learning (Shen et al, 2019) or zero-shot learning has also been studied (Shah et al, 2019). To the best of our knowledge, none of the existing work introduces a multi-task learning solely for the SF to incorporate the contextual information in both representation and task levels.…”

Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU

“…Shen et al (2019b,a) introduce a SkillBot that allows users to build up their own skills. Recently, Ray et al (2018Ray et al ( , 2019; Shen et al (2018bShen et al ( , 2019d enables an SLU model to incorporate user personalization over time. However, the above approaches do not explicitly address unsupported user utterances/intents, leading to catastrophic failures illustrated in Figure 1.…”