ChatGPT as a Factual Inconsistency Evaluator for Abstractive Text Summarization

Luo, Zheheng; Ananiadou, Sophia

doi:10.48550/arxiv.2303.15621

Cited by 21 publications

(28 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In existing LLM-based text annotation methods [2,33,37,48], several prompt-tuning techniques have been utilized to enhance the performance of LLMs. Alizadeh et al [2] investigate the performance difference between LLMs and crowd-workers in text annotation tasks.…”

Section: Prompt-tuning For Text Annotationmentioning

confidence: 99%

APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT

Zhu,

Yin,

Tyson

et al. 2024

Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

Recent research has highlighted the potential of LLMs, like Chat-GPT, for performing label annotation on social computing text. However, it is already well known that performance hinges on the quality of the input prompts. To address this, there has been a flurry of research into prompt tuning -techniques and guidelines that attempt to improve the quality of prompts. Yet these largely rely on manual effort and prior knowledge of the dataset being annotated. To address this limitation, we propose APT-Pipe, an automated prompt-tuning pipeline. APT-Pipe aims to automatically tune prompts to enhance ChatGPT's text classification performance on any given dataset. We implement APT-Pipe and test it across twelve distinct text classification datasets. We find that prompts tuned by APT-Pipe help ChatGPT achieve higher weighted F1-score on nine out of twelve experimented datasets, with an improvement of 7.01% on average. We further highlight APT-Pipe's flexibility as a framework by showing how it can be extended to support additional tuning mechanisms. CCS CONCEPTS• Computing methodologies → Natural language processing.

show abstract

Section: Prompt-tuning For Text Annotationmentioning

confidence: 99%

APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT

Zhu,

Yin,

Tyson

et al. 2024

Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

show abstract

“…Yang et al [34] explored the limitations of ChatGPT in query-based summarization. Luo et al [35] studied the practicality of ChatGPT as a fact-inconsistency evaluator for abstract text summarization. Recently, COT has significantly improved the inferential performance and interpretability of LLMs by decomposing multi-step problems into intermediate steps [20,36,37].…”

Section: Large Langauge Modelsmentioning

confidence: 99%

3A-COT: An Attend-Arrange-Abstract Chain-of-Thought for Multi-Document Summarization

Zhang,

Gao,

Huang

et al. 2024

Preprint

View full text Add to dashboard Cite

Large language models (LLMs) have shown significant promise in single-document summarization (SDS). However, there has been limited exploration in multi-document summarization (MDS). The primary challenge in MDS lies in effectively navigating the intricate and complex relationships between multiple documents, including issues such as contradictions, redundancies, and complementarities. Taking inspiration from chain-of-thought (CoT) and how human approaches to MDS, this paper proposes a novel MDS method called Attend-Arrange-Abstract CoT (3A-COT). The method comprises three critical stages. Attend: To alleviate redundancies between multiple documents, key information should be focused first. Therefore, we design an Attend-prompt to extract key elements from each document and generate key information based on these key elements. Arrange: Considering the contradictions and complementarities between these key information, we design an Arrange-prompt to arrange the relationships and connections between these key information across multiple documents to form reasonable logic relationships. Abstract: Finally, an Abstract-prompt is designed to guide the generation of a summary based on the arranged key information. To evaluate 3A-COT, we conduct experiments on four MDS test sets: Multi-News, Human-edited Multi-News, WCEP-10, and Multi-XScience. Experimental results show that 3A-COT outperforms other LLM-based MDS methods. Furthermore, in-depth analysis experiments are conducted to evaluate 3A-COT's efficacy in handling contradictions, redundancies, and complementarities.

show abstract

“…The study sought to provide a valuable contribution to the future progress of chatbot and language model development. Zheheng Luo et al examined the problems regarding factual inconsistencies [4] in generated summaries by utilizing large language models (LLMs). It specifically investigated the evaluation capabilities of ChatGPT under a zeroshot environment.…”

Section: Review On the Limitations Of Chatgptmentioning

confidence: 99%