Exploring ChatGPT for next-generation information retrieval: Opportunities and challenges

Background: Machine learning (ML) promises versatile help in the creation of systematic reviews (SRs). Recently, further developments in the form of large language models (LLMs) and their application in SR conduct attracted attention. Objective: To provide an overview of ML and specifically LLM applications in SR conduct in health research. Study design: We systematically searched MEDLINE, Web of Science, IEEEXplore, ACM Digital Library, Europe PMC (preprints), Google Scholar, and conducted an additional hand search (last search: 26 February 2024). We included scientific articles in English or German, published from April 2021 onwards, building upon the results of a mapping review with a related research question. Two reviewers independently screened studies for eligibility; after piloting, one reviewer extracted data, checked by another. Results: Our database search yielded 8054 hits, and we identified 33 articles from our hand search. Of the 196 included reports, 159 described more traditional ML techniques, 37 focused on LLMs. LLM approaches covered 10 of 13 defined SR steps, most frequently literature search (n=15, 41%), study selection (n=14, 38%), and data extraction (n=11, 30%). The mostly recurring LLM was GPT (n=33, 89%). Validation studies were predominant (n=21, 57%). In half of the studies, authors evaluated LLM use as promising (n=20, 54%), one quarter as neutral (n=9, 24%) and one fifth as non-promising (n=8, 22%). Conclusions: Although LLMs show promise in supporting SR creation, fully established or validated applications are often lacking. The rapid increase in research on LLMs for evidence synthesis production highlights their growing relevance.

show abstract

A study of the impact of ChatGPT self-efficacy on the information seeking behaviors in ChatGPT: the mediating roles of ChatGPT characteristics and utility

Mun

2024

OIR

View full text Add to dashboard Cite

PurposeThe purpose of this study was to examine the factors that influence the information seeking behaviors of ChatGPT users. Specifically, we investigated how ChatGPT self-efficacy, ChatGPT characteristics and ChatGPT utility affect the frequency and duration of information seeking via ChatGPT. We also tested the mediating roles of ChatGPT characteristics and utility in the relationship between ChatGPT self-efficacy and information-seeking behaviors.Design/methodology/approachThis study adopts a quantitative approach and collects data from 403 ChatGPT users using an online questionnaire. The data are analyzed using linear regression and structural equation modeling (SEM).FindingsThe linear regression analyses revealed that ChatGPT self-efficacy is positively and significantly related to the information seeking behaviors in ChatGPT. Second, mediation analyses also showed that ChatGPT characteristics and utility significantly mediate the relationship between ChatGPT self-efficacy and information-seeking behaviors in ChatGPT independently and sequentially.Originality/valueThis study is the first to investigate the factors and mechanisms that influence information-seeking behaviors in ChatGPT, a new phenomenon in the media landscape. The findings in this study suggest that ChatGPT self-efficacy acts as an important motivator for information-seeking behaviors in ChatGPT and that ChatGPT characteristics and utility provide information regarding potential mechanisms in the relationship between ChatGPT self-efficacy and information-seeking behaviors in ChatGPT. The study contributes to the literature on information seeking, self-efficacy and generative AI.

show abstract

Exploring ChatGPT for next-generation information retrieval: Opportunities and challenges

Cited by 4 publications

References 55 publications

Situational Data Integration in Question Answering systems: a survey over two decades

Situational Data Integration in Question Answering systems: a survey over two decades

Large language models for conducting systematic reviews: on the rise, but not yet ready for use – a scoping review

A study of the impact of ChatGPT self-efficacy on the information seeking behaviors in ChatGPT: the mediating roles of ChatGPT characteristics and utility

Contact Info

Product

Resources

About