Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions

Alshami, Ahmad; Elsayed, Moustafa; Ali, Eslam; Eltoukhy, Abdelrahman E. E.; Zayed, Tarek

doi:10.3390/systems11070351

Cited by 74 publications

(49 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we prompted the LLM GPT-4 to answer specific questions about entire papers whereas previous studies were often optimized for screening paper title and abstracts (5)(6)(7)(8). Second, we used GPT without providing training examples, whereas previous studies, were often interactive in that they combined NLP and ML algorithms with user feedback (5,6,(9)(10)(11). Finally, our results were quantitative and transparent, whereas several previous studies, particularly those using LLMs, presented their results in a qualitative manner.…”

Section: Discussionmentioning

confidence: 94%

“…Most such tools have used natural language processing (NLP) and machine learning (ML) algorithms primarily to screen the titles and abstracts of publications to determine whether they meet the search criteria for a systematic review (2)(3)(4)(5)(6)(7)(8). Several studies have also described the potential for using the representational language model BERT (Bidirectional Encoder Representations from Transformers) and the Generative Pre-trained Transformer (GPT) large language models (LLMs) for reviewing the full text of published studies (9)(10)(11)(12). LLMs have also been evaluated for their ability to summarize research studies (13,14).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GPT-4 Performance on Querying Scientific Publications: Reproducibility, Accuracy, and Impact of an Instruction Sheet

Tao,

Osman,

Tzou

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: Large language models (LLMs) that could efficiently screen and identify studies fulfilling specific criteria, as well as those capable of data extraction from publications, would streamline literature reviews and enhance knowledge discovery by lessening the burden on human reviewers. Methods: We created an automated pipeline utilizing OpenAI GPT-4 32K API version “2023-05-15” to evaluate the accuracy of the LLM GPT-4 when responding to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in PubMed. We presented the 60 papers to GPT-4 in four configurations: (1) all 60 questions simultaneously; (2) all 60 questions simultaneously with the instruction sheet; (3) each of the 60 questions individually; and (4) each of the 60 questions individually with the instruction sheet. Results: GPT-4 achieved a mean accuracy of 86.9% – 24.0% higher than when the answers to papers were permuted. The overall recall and precision were 72.5% and 87.4%, respectively. The standard deviation of three replicates for the 60 questions ranged from 0 to 5.3% with a median of 1.2%. The instruction sheet did not significantly increase GPT-4’s accuracy, recall, or precision. GPT-4 was more likely to provide false positive answers when the 60 questions were submitted individually compared to when they were submitted together. Conclusions: GPT-4 reproducibly answered 3600 questions about 60 papers on HIVDR with moderately high accuracy, recall, and precision. The inability of the instruction sheet to increase these metrics suggests that more sophisticated prompt engineering approaches or the finetuning of an open source model are required to further improve the ability of an LLM to answer questions about highly specialized HIV drug resistance papers.

show abstract

Section: Discussionmentioning

confidence: 94%

Section: Introductionmentioning

confidence: 99%

GPT-4 Performance on Querying Scientific Publications: Reproducibility, Accuracy, and Impact of an Instruction Sheet

Tao,

Osman,

Tzou

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…[ 96–99 ] While they are not designed to replace human experts with specialized domain knowledge, these models could significantly streamline and simplify the process of conducting literature reviews. [ 98,100 ] This could become an invaluable tool for researchers trying to understand a field and identify emerging trends and key discoveries within it.…”

Section: The Third Rung: Generative Models and Hypothesis Generationmentioning

confidence: 99%

The Future of Material Scientists in an Age of Artificial Intelligence

Maqsood,

Chen,

Jacobsson

2024

Advanced Science

View full text Add to dashboard Cite

Material science has historically evolved in tandem with advancements in technologies for characterization, synthesis, and computation. Another type of technology to add to this mix is machine learning (ML) and artificial intelligence (AI). Now increasingly sophisticated AI‐models are seen that can solve progressively harder problems across a variety of fields. From a material science perspective, it is indisputable that machine learning and artificial intelligence offer a potent toolkit with the potential to substantially accelerate research efforts in areas such as the development and discovery of new functional materials. Less clear is how to best harness this development, what new skill sets will be required, and how it may affect established research practices. In this paper, those question are explored with respect to increasingly more sophisticated ML/AI‐approaches. To structure the discussion, a conceptual framework of an AI‐ladder is introduced. This AI‐ladder ranges from basic data‐fitting techniques to more advanced functionalities such as semi‐autonomous experimentation, experimental design, knowledge generation, hypothesis formulation, and the orchestration of specialized AI modules as stepping‐stones toward general artificial intelligence. This ladder metaphor provides a hierarchical framework for contemplating the opportunities, challenges, and evolving skill sets required to stay competitive in the age of artificial intelligence.

show abstract

“…There are some initial attempts to evaluate ChatGPTs in automated SR, such as automating search queries [49]. Alshami et al [50] leveraged GPT to automate the SR pipeline, yet their methodology diverges from conventional abstract screening practices, rendering it distinct from traditional approaches. The application of GPT in abstract screening has been scarcely explored.…”

Section: Background Studymentioning

confidence: 99%

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

Akinseloyin,

Jiang,

Palade

2023

Preprint

View full text Add to dashboard Cite

ObjectiveThis paper aims to address the challenges in citation screening (a.k.a. abstract screening) within Systematic Reviews (SR) by leveraging the zero-shot capabilities of large language models, particularly ChatGPT.MethodsWe employ ChatGPT as a zero-shot ranker to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Citation screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by ChatGPT. The framework involves breaking down the selection criteria into multiple questions, properly prompting ChatGPT to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions.ResultsLarge-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology Assisted Reviews in Empirical Medicine. Across 31 datasets of four categories of SRs, the proposed QA framework consistently outperformed other zero-shot ranking models. Compared with complex ranking approaches with iterative relevance feedback and fine-tuned deep learning-based ranking models, our ChatGPT-based zero-shot citation screening approaches still demonstrated competitive and sometimes better results, underscoring their high potential in facilitating automated systematic reviews.ConclusionInvestigation justified the indispensable value of leveraging selection criteria to improve the performance of automated citation screening. ChatGPT demonstrated proficiency in prioritizing candidate studies for citation screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance citation screening.

show abstract

Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions

Cited by 74 publications

References 51 publications

GPT-4 Performance on Querying Scientific Publications: Reproducibility, Accuracy, and Impact of an Instruction Sheet

GPT-4 Performance on Querying Scientific Publications: Reproducibility, Accuracy, and Impact of an Instruction Sheet

The Future of Material Scientists in an Age of Artificial Intelligence

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

Contact Info

Product

Resources

About