Human-AI Collaboration to Identify Literature for Evidence Synthesis

Abstract

Systematic approaches to evidence synthesis can improve the rigour, transparency, and replicability of a traditional literature review. However, these systematic approaches are time and resource intensive. We evaluate the ability of OpenAI’s ChatGPT to undertake two initial stages of evidence syntheses (searching peer-reviewed literature and screening for relevance) and develop a novel collaborative framework to leverage the best of both human and AI intelligence. Using a scoping review of community-based fisheries management as a case study, we find that with substantial prompting, the AI can provide critical insight into the construction and content of a search string. Thereafter, we evaluate five strategies for synthesising AI output to screen articles based on predefined inclusion criteria. We find low omission rates (< 1%) of relevant literature by the AI are achievable, which is comparable to that of human screeners. These findings show that generalised AI tools can assist reviewers with evidence synthesis to accelerate the implementation and improve the reliability of a review.

Leveraging AI to improve evidence synthesis in conservation

“…In the medical sciences, where evidence synthesis methods are well developed and widely used, recent studies demonstrate the promising role that AI tools can play in carrying out rapid and extensive literature reviews [8,12]. At the same time, there is also discourse around potential challenges and limitations regarding the usefulness of these platforms [7,[13][14][15].…”

“…Researchers have shown that the use of LLM tools can substantially shorten, by as much as six-fold, the time spent screening relevant research ( [8,12,13], Box 2). LLMs could also be applied to (meta)data extraction from relevant studies and summarize a collection of articles more efficiently [8,16,17].…”

“…Using LLM tools can assist in reducing these human biases. For example, by improving prompts, Spillias et al ( [13]) were able to increase the accuracy of screening with ChatGPT (reducing type II errors to < 1%). By helping locate potentially useful gray literature sources, which can be a critical source of biodiversity conservation evidence [20,21], LMMs can help further reduce the effects of publication bias on review comprehensiveness, and can act as a second or third nonhuman reviewer to tackle screening inconsistencies [13].…”

“…By translating languages, AI tools can make all stages of the review process more inclusive (Box 1). For example, a review on community-based fisheries management focusing on the Pacific Islands [13] benefited from AI rapidly providing a list of non-English relevant terms to be integrated into the search string and yielding additional articles not previously identified by the original search. AI-suggested terms should, however, be checked by proficient speakers of the language in question before inclusion in the search string.…”

“…There are a number of AI-assisted article screening tools, most of which use re-ordering algorithms that learn from included/excluded articles as researchers screen based on title and abstract. More recently, large language models (LLMs) have been suggested to be used for such screening [13]. We tested both types: Rayyan.ai (re-ordering algorithm) and GPT 3.…”