Document-Grounded Goal-Oriented Dialogue Systems on Pre-Trained Language Model with Diverse Input Representation

Abstract

Document-grounded goal-oriented dialog system understands users' utterances, and generates proper responses by using information obtained from documents. The Dialdoc21 shared task consists of two subtasks; subtask1, finding text spans associated with users' utterances from documents, and subtask2, generating responses based on information obtained from subtask1. In this paper, we propose two models (i.e., a knowledge span prediction model and a response generation model) for the sub-task1 and the subtask2. In the subtask1, dialogue act losses are used with RoBERTa, and title embeddings are added to input representation of RoBERTa. In the subtask2, various special tokens and embeddings are added to input representation of BART's encoder. Then, we propose a method to assign different difficulty scores to leverage curriculum learning. In the subtask1, our span prediction model achieved F1-scores of 74.81 (ranked at top 7) and 73.41 (ranked at top 5) in test-dev phase and test phase, respectively. In the subtask2, our response generation model achieved sacre-BLEUs of 37.50 (ranked at top 3) and 41.06 (ranked at top 1) in in test-dev phase and test phase, respectively.

UniGDD: A Unified Generative Framework for Goal-Oriented Document-Grounded Dialogue

“…Response generation then aims at generating a proper agent response according to the dialogue context and the selected knowledge. Therefore, one straightforward solution for this problem is to use two models to conduct KI and RG in a pipeline manner (Daheim et al, 2021;Kim et al, 2021;Xu et al, 2021;Chen et al, 2021;. However, such pipeline methods fail to capture the interdependence between KI and RG.…”

Curricular transfer learning for complex grammar acquisition in language models

“…Most of them are based on pre-trained LMs. For example, Kim et al (2021) used a RoBERTa and BART model to predict slot values and generate responses, respectively. Their study employed a teacher model to score examples (using F1-score and BLEU) from easier to harder.…”

“…These easier examples are not restricted to costly data acquisition, e.g., our costs were marginally none. (Bengio et al, 2009) x (Pentina et al, 2015) x (Narvekar et al, 2016) x (Saito, 2018) x (Fang et al, 2019) x (Foglino and Leonetti, 2019) x x x (Kim et al, 2021) x (Dai et al, 2021) x (Zhu et al, 2021b) x (Zhao et al, 2022) x CTL (ours)…”