Overview of the CLEF Dynamic Search Evaluation Lab 2018

Kanoulas, Evangelos; Azzopardi, Leif; Yang, Grace Hui

doi:10.1007/978-3-319-98932-7_31

Cited by 9 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiments are conducted using two collections: CLEF technological assisted reviews (TAR) datasets [28][29][30] and systematic review collection with seed studies (Seed Collection) [79]. CLEF TAR dataset is published each year from 2017 to 2019 as a validation dataset for more effective systematic review Boolean query formulation and screening.…”

Section: Experimental Settingsmentioning

confidence: 99%

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Wang¹,

Scells²,

Koopman³

et al. 2023

Preprint

View full text Add to dashboard Cite

Systematic reviews are comprehensive reviews of the literature for a highly focused research question. These reviews are often treated as the highest form of evidence in evidence-based medicine, and are the key strategy to answer research questions in the medical field. To create a high-quality systematic review, complex Boolean queries are often constructed to retrieve studies for the review topic. However, it often takes a long time for systematic review researchers to construct a high quality systematic review Boolean query, and often the resulting queries are far from effective. Poor queries may lead to biased or invalid reviews, because they missed to retrieve key evidence, or to extensive increase in review costs, because they retrieved too many irrelevant studies. Recent advances in Transformer-based generative models have shown great potential to effectively follow instructions from users and generate answers based on the instructions being made. In this paper, we investigate the effectiveness of the latest of such models, ChatGPT, in generating effective Boolean queries for systematic review literature search. Through a number of extensive experiments on standard test collections for the task, we find that ChatGPT is capable of generating queries that lead to high search precision, although trading-off this for recall. Overall, our study demonstrates the potential of ChatGPT in generating effective Boolean queries for systematic review literature search. The ability of ChatGPT to follow complex instructions and generate queries with high precision makes it a valuable tool for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and often trading-off higher precision for lower recall is acceptable.

show abstract

Section: Experimental Settingsmentioning

confidence: 99%

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Wang¹,

Scells²,

Koopman³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The 2018 Lab focuses on the development of an evaluation framework, where participants submit "querying agents" that generate queries to be submitted to a static retrieval system. Effective "querying agents" can then simulate users towards developing dynamic search systems [10].…”

Section: The Clef Lab Sessionsmentioning

confidence: 99%

Report on CLEF 2018

et al. 2019

View full text Add to dashboard Cite

This is a report on the ninth edition of the Conference and Labs of the Evaluation Forum (CLEF 2018), held in early September 2018, in Avignon, France. CLEF was a four day event combining a Conference and an Evaluation Forum. The Conference featured keynotes by Nicholas Belkin, Julio Gonzalo, and Gabriella Pasi, and presentation of 29 peer reviewed research papers covering a wide range of topics in addition to many posters. The Evaluation Forum consisted to ten Labs: CENTRE, CheckThat, DynSe, eRisk, eHealth, ImageCLEF, LifeCLEF, Cultural Microblog Contextualization, PAN, and PIR-CLEF, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.

show abstract

“…Methods to automate such review processes were evaluated in the scope of the Conference and Labs of Evaluation Forum (CLEF) with the so-called eHealth challenges regarding Technology Assisted Reviews (TAR) for systematic reviews (SR) in Empirical Medicine [10], in which relevant documents must be automatically retrieved for a given topic. The bestperforming method achieved an almost perfect overall recall of relevant documents, while the recall regarding the first search results and consequently the workload reduction could be optimized further [11].…”

Section: Introductionmentioning

confidence: 99%

Guidance for Clinical Evaluation under the Medical Device Regulation through Automated Scoping Searches

et al. 2023

View full text Add to dashboard Cite

The Medical Device Regulation (MDR) in Europe aims to improve patient safety by increasing requirements, particularly for the clinical evaluation of medical devices. Before the clinical evaluation is initiated, a first literature review of existing clinical knowledge is necessary to decide how to proceed. However, small and medium-sized enterprises (SMEs) lacking the required expertise and funds may disappear from the market. Automating searches for the first literature review is both possible and necessary to accelerate the process and reduce the required resources. As a contribution to the prevention of the disappearance of SMEs and respective medical devices, we developed and tested two automated search methods with two SMEs, leveraging Medical Subject Headings (MeSH) terms and Bidirectional Encoder Representations from Transformers (BERT). Both methods were tailored to the SMEs and evaluated through a newly developed workflow that incorporated feedback resource-efficiently. Via a second evaluation with the established CLEF 2018 eHealth TAR dataset, the more general suitability of the search methods for retrieving relevant data was tested. In the real-world use case setting, the BERT-based method performed better with an average precision of 73.3%, while in the CLEF 2018 eHealth TAR evaluation, the MeSH-based search method performed better with a recall of 86.4%. Results indicate the potential of automated searches to provide device-specific relevant data from multiple databases while screening fewer documents than in manual literature searches.

show abstract

Overview of the CLEF Dynamic Search Evaluation Lab 2018

Cited by 9 publications

References 17 publications

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Report on CLEF 2018

Guidance for Clinical Evaluation under the Medical Device Regulation through Automated Scoping Searches

Contact Info

Product

Resources

About