Prompt scoring system for dialogue summarization using GPT-3

Abstract

<p>Recent results in language processing show that language models are capable of performing several natural language tasks without the need of supervised learning. A challenging task for pre-trained language models is dialogue summarization. One way of generating summaries is engineering prompt templates for few-shot training. However, a static approach of creating prompts leads to unreliable outcomes between different classes of dialogues. Focusing on the dialogues structure properties, we propose a scoring system to improve the few-shot training performances by building tuned prompts composed by the highest scored dialogue samples. Our evaluation based on ROUGE scores and human evaluation shows that there is improvement for the experiments where we made use of the score system. The positive results are validated by all the three large-scale datasets we used in testing. All experiments were performed within the framework of the GPT-3 API.</p>

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

“…Research works exploring GLLMs for dialogue tasks. The research community explored GLLMs like GPT-3, GPT-3.5 and ChatGPT for various dialogue tasks like dialogue summarization [157], [220], [221] , dialogue question answering [224], emotion dialogue understanding and generation [219], dialogue state tracking [218], dialogue generation [132], and dialogue discourse analysis [223]. Some of the research works explored LLMs for the evaluation of dialogue tasks [222].…”

“…Prodan et al [221] proposed a scoring system to choose the best examples for dialogue summarizing using few-shot GPT-3. The proposed scoring system enhances the quality of generated summaries with an 11% reduction in failures.…”