Transferring knowledge between topics in systematic reviews

Molinari, Alessio; Kanoulas, Evangelos

doi:10.1016/j.iswa.2022.200150

Cited by 6 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Appendix A in the online supplementary material contains detailed statistics of the SRs in each category. We refrained from using the training set provided by TAR2019, aiming to highlight the effectiveness of our zero-shot methodology that eliminates the need for prior training [64, 23]. Selection criteria for each SR are included in TAR2019.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

Akinseloyin,

Jiang,

Palade

2023

Preprint

View full text Add to dashboard Cite

ObjectiveThis paper aims to address the challenges in citation screening (a.k.a. abstract screening) within Systematic Reviews (SR) by leveraging the zero-shot capabilities of large language models, particularly ChatGPT.MethodsWe employ ChatGPT as a zero-shot ranker to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Citation screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by ChatGPT. The framework involves breaking down the selection criteria into multiple questions, properly prompting ChatGPT to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions.ResultsLarge-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology Assisted Reviews in Empirical Medicine. Across 31 datasets of four categories of SRs, the proposed QA framework consistently outperformed other zero-shot ranking models. Compared with complex ranking approaches with iterative relevance feedback and fine-tuned deep learning-based ranking models, our ChatGPT-based zero-shot citation screening approaches still demonstrated competitive and sometimes better results, underscoring their high potential in facilitating automated systematic reviews.ConclusionInvestigation justified the indispensable value of leveraging selection criteria to improve the performance of automated citation screening. ChatGPT demonstrated proficiency in prioritizing candidate studies for citation screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance citation screening.

show abstract

Section: Methodsmentioning

confidence: 99%

“…We employed the TAR2019 test set comprising 31 SRs categorized into Intervention (20), DTA (8), Qualitative (2), and Prognosis (1). We refrained from using the training set, aiming to highlight the effectiveness of our zero-shot methodology that eliminates the need for prior training [65,25].…”

Section: Dataset and Evaluationmentioning

confidence: 99%

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

Akinseloyin,

Jiang,

Palade

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Most work on this task has focused on the use of traditional machine learning models such as SVM [4,52] and logistic regression [28,36,46]. Recent studies have shown the potential of BERT-based models for this task [35,45,49]. Many techniques are also leveraged together with these models, such as active learning [10,11,18,28,37,42,52], and relevance feedback [4,10,11,21,52], which is also seen in combination with query-based methods [1,3,[12][13][14].…”

Section: Introductionmentioning

confidence: 99%

Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation

Mao,

Zhuang,

Koopman

et al. 2024

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

The goal of screening prioritisation in systematic reviews is to identify relevant documents with high recall and rank them in early positions for review. This saves reviewing effort if paired with a stopping criterion, and speeds up review completion if performed alongside downstream tasks. Recent studies have shown that neural models have good potential on this task, but their time-consuming finetuning and inference discourage their widespread use for screening prioritisation. In this paper, we propose an alternative approach that still relies on neural models, but leverages dense representations and relevance feedback to enhance screening prioritisation, without the need for costly model fine-tuning and inference. This method exploits continuous relevance feedback from reviewers during document screening to efficiently update the dense query representation, which is then applied to rank the remaining documents to be screened. We evaluate this approach across the CLEF TAR datasets for this task. Results suggest that the investigated dense query-driven approach is more efficient than directly using neural models and shows promising effectiveness compared to previous methods developed on the considered datasets. Our code is available at https://github.com/ielab/dense-screening-feedback. CCS CONCEPTS• Information systems → Information retrieval.

show abstract

“…Moreover, it was found that for studies that use more than four datasets [4,7,9,11,16,18,19,21,24,25,23,31,32,38,52,64,65,68,30], their domain is predominantly restricted to medicine reviews, as seen in Figure 1. Even the high quality Cohen dataset [8], known as the gold standard in the field, is solely restricted to drug class reviews.…”

Section: Introductionmentioning

confidence: 99%

Large-Scale Simulation Study of Active Learning Models for Systematic Reviews

Teijema,

de Bruin,

Bagheri

et al. 2023

Preprint

View full text Add to dashboard Cite

The active learning methods for prioritising systematic reviews have undergone significant progress and innovation in recent years. This rapid development, however, has inadvertently highlighted the disparity between the rapid development of these methodologies and their rigorous evaluation, stemming from constraints in simulation size, lack of infrastructure, and the use of few datasets.We embark on a large-scale simulation study involving over 27 thousand simulations and over 156 million datapoints, designed to provide robust empirical evidence of active learning methodologies performance. We evaluate 13 combinations of different classification models and feature extraction techniques across high-quality datasets sourced from the SYNERGY dataset. We run a single simulation for each possible combination of selected classification model, feature extraction technique, dataset, and relevant document. The spectrum of performance varies considerably, from marginally better than random reading to near flawless results. Still, every single model-feature extraction combination outperforms random screening. Results are publicly available for analysis and replication.This study advocates for large-scale simulations as the gold standard for assessing active learning methods; it underscores the importance of comprehensive testing to reduce reporting bias and enhance result reliability. It also highlights the need for curating diverse datasets for systematic review literature.

show abstract

Transferring knowledge between topics in systematic reviews

Cited by 6 publications

References 14 publications

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation

Large-Scale Simulation Study of Active Learning Models for Systematic Reviews

Contact Info

Product

Resources

About