Enhancing English abstract quality for non-English speaking authors using ChatGPT: A comparative study of Taiwan, Japan, China, and South Korea with slope graphs

Abstract

A clear and proficient English abstract is crucial for disseminating research findings to a global audience, significantly impacting the accessibility and visibility of research from non-English speaking countries. Despite the adoption of ChatGPT since November 30, 2022, a comprehensive analysis of improvements in English abstracts in scholarly journals has not been conducted. This study aims to identify which authors from Taiwan, Japan, China, and South Korea (TJCS) have shown the most improvement in English abstracts. Article abstracts published in Medicine (Baltimore) sourced from the Web of Science Core Collection from 2020 to 2023 were downloaded. A mixed-methods approach was employed, combining quantitative analysis of linguistic quality indicators and qualitative assessments of coherence and engagement using the Rasch model. Ten quality indicators were determined by prompting ChatGPT. Two scenarios were analyzed: (1) generative pretrained transformer (GPT) versus non-GPT (each with 30 abstracts from 2021) and (2) TJCS in comparison (each with 100 abstracts from 2021 and 2023, respectively). Standardized mean differences were compared using paired samples t test. Visuals including forest plots, Rasch Wright Map, the slope graph, and scatter plot with 95% control lines were used to examine the 2 scenarios. (1) No significant difference was found between GPT and non-GPT abstracts with Rasch logit scores of 3.31 and 3.17, respectively (P = .42), likely due to small sample size (n = 30); (2) significant difference exists between 2020 and 2023 in each country, and between South Korea and Taiwan in 2020. Among TJCS, Taiwan showed the greatest improvement in English abstract quality post-ChatGPT implementation, followed by Japan, China, and South Korea. The English abstracts in Medicine (Baltimore) have improved, reflecting the tool’s positive impact on enhancing technical language. This study demonstrates that ChatGPT can enhance the quality of English abstracts for authors from non-English speaking regions, although the assumption that all authors use ChatGPT is invalid and impractical. The findings underscore the value of artificial intelligence tools in academic writing and recommend further investigation into the long-term implications of artificial intelligence integration in scholarly communication.