Shades of BLEU, Flavours of Success: The Case of MultiWOZ

Nekvinda, Tomáš; Dušek, Ondřej

doi:10.48550/arxiv.2106.05555

Cited by 2 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluated the performance of PEFTTOD in the context of task-oriented dialogue systems for end-to-end dialogue modeling [45]. The evaluation was conducted using the benchmark dataset Multi-WOZ 2.0 [12].…”

Section: Discussionmentioning

confidence: 99%

“…PEFTTOD was trained on the Multi-WOZ 2.0 dataset, specifically on the task of the end-to-end dialogue modeling [45]. The proposed system was trained using the maximum likelihood method, a common approach in machine learning, which aims to optimize the model's parameters by maximizing the likelihood of generating the correct outputs given the inputs.…”

Section: End-to-end Dialogue Modelingmentioning

confidence: 99%

See 1 more Smart Citation

Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems

Yoo

Kang

2023

Mathematics

View full text Add to dashboard Cite

The use of Transformer-based pre-trained language models has become prevalent in enhancing the performance of task-oriented dialogue systems. These models, which are pre-trained on large text data to grasp the language syntax and semantics, fine-tune the entire parameter set according to a specific task. However, as the scale of the pre-trained language model increases, several challenges arise during the fine-tuning process. For example, the training time escalates as the model scale grows, since the complete parameter set needs to be trained. Furthermore, additional storage space is required to accommodate the larger model size. To address these challenges, we propose a new new task-oriented dialogue system called PEFTTOD. Our proposal leverages a method called the Parameter-Efficient Fine-Tuning method (PEFT), which incorporates an Adapter Layer and prefix tuning into the pre-trained language model. It significantly reduces the overall parameter count used during training and efficiently transfers the dialogue knowledge. We evaluated the performance of PEFTTOD on the Multi-WOZ 2.0 dataset, a benchmark dataset commonly used in task-oriented dialogue systems. Compared to the traditional method, PEFTTOD utilizes only about 4% of the parameters for training, resulting in a 4% improvement in the combined score compared to the existing T5-based baseline. Moreover, PEFTTOD achieved an efficiency gain by reducing the training time by 20% and saving up to 95% of the required storage space.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: End-to-end Dialogue Modelingmentioning

confidence: 99%

Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems

Yoo

Kang

2023

Mathematics

View full text Add to dashboard Cite

show abstract

“…Despite the availability of toolkits that facilitate user simulation (US) evaluation (Zhu et al, 2020), corpus-based match and success rates are the default benchmark for works in task-oriented dialogue systems today (Budzianowski et al, 2018;Nekvinda and Dušek, 2021). These metrics are practical to compute, reproducible, and scalable.…”

Section: Related Workmentioning

confidence: 99%

Dialogue Evaluation with Offline Reinforcement Learning

Lubis,

Geishauser,

Lin

et al. 2022

Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

Task-oriented dialogue systems aim to fulfill user goals through natural language interactions. They are ideally evaluated with human users, which however is unattainable to do at every iteration of the development phase. Simulated users could be an alternative, however their development is nontrivial. Therefore, researchers resort to offline metrics on existing human-human corpora, which are more practical and easily reproducible. They are unfortunately limited in reflecting real performance of dialogue systems. BLEU for instance is poorly correlated with human judgment, and existing corpus-based metrics such as success rate overlook dialogue context mismatches. There is still a need for a reliable metric for taskoriented systems with good generalization and strong correlation with human judgements. In this paper, we propose the use of offline reinforcement learning for dialogue evaluation based on a static corpus. Such an evaluator is typically called a critic and utilized for policy optimization. We go one step further and show that offline RL critics can be trained the static corpus for any dialogue system as external evaluators, allowing dialogue performance comparisons across various types of systems. This approach has the benefit of being corpusand model-independent, while attaining strong correlation with human judgements, which we confirm via an interactive user trial.

show abstract

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

Cited by 2 publications

References 9 publications

Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems

Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems

Dialogue Evaluation with Offline Reinforcement Learning

Contact Info

Product

Resources

About