Impact of a Search Engine on Clinical Decisions Under Time and System Effectiveness Constraints: Research Protocol

Abstract

Background Many clinical questions arise during patient encounters that clinicians are unable to answer. An evidence-based medicine approach expects that clinicians will seek and apply the best available evidence to answer clinical questions. One commonly used source of such evidence is scientific literature, such as that available through MEDLINE and PubMed. Clinicians report that 2 key reasons why they do not use search systems to answer questions is that it takes too much time and that they do not expect to find a definitive answer. So, the question remains about how effectively scientific literature search systems support time-pressured clinicians in making better clinical decisions. The results of this study are important because they can help clinicians and health care organizations to better assess their needs with respect to clinical decision support (CDS) systems and evidence sources. The results and data captured will contribute a significant data collection to inform the design of future CDS systems to better meet the needs of time-pressured, practicing clinicians. Objective The purpose of this study is to understand the impact of using a scientific medical literature search system on clinical decision making. Furthermore, to understand the impact of realistic time pressures on clinicians, we vary the search time available to find clinical answers. Finally, we assess the impact of improvements in search system effectiveness on the same clinical decisions. Methods In this study, 96 practicing clinicians and final year medical students are presented with 16 clinical questions which they must answer without access to any external resource. The same questions are then represented to the clinicians; however, in this part of the study, the clinicians can use a scientific literature search engine to find evidence to support their answers. The time pressures of practicing clinicians are simulated by limiting answer time to one of 3, 6, or 9 min per question. The correct answer rate is reported both before and after search to assess the impact of the search system and the time constraint. In addition, 2 search systems that use the same user interface, but which vary widely in their search effectiveness, are employed so that the impact of changes in search system effectiveness on clinical decision making can also be assessed. Results Recruiting began for the study in June 2018. As of the April 4, 2019, there were 69 participants enrolled. The study is expected to close by May 30, 2019, with results to be published in July. Conclusions All data collected in this study will be made available at the University of Queensland’s UQ eSpace public data repository. International Registered Report Identifier (IRRID) DERR1-10.2196/12803

How searching under time pressure impacts clinical decision making

“…The detailed study protocol was previously reported by van der Vegt et al [20]. Several preliminary trials related to question selection and study timing were conducted with clinicians prior to finalizing the protocol to ensure that participants could successfully complete the study.…”

“…We developed a custom, experimental medical literature search system instead of using an existing search engine, such as PubMed, to ensure full control of the system and because the use of an existing search engine would have exposed participants to a changing document collection over the study time period. The quality of the search system was assessed using standard offline evaluation practices [ 28 ] and represents a state-of-the-art clinical literature search [ 20 ].…”

“…Answer=Yes; Source evidence PubMed IDs= [3417926,3459718] A full listing of questions and answers can be found in van der Vegt et al [20].…”

“…To assess the impact of time pressure on clinical decision accuracy, participants' answers in both stage 1 and 2 were coded as right or wrong by comparing their answers to an expert-judged assessment of each question (provided together with the scenarios in van der Vegt et al [20]). Stage 2 questions for which no search was performed were discarded, as the value of the search system could not be confirmed in these cases.…”

Computer-Assisted Relevance Assessment: A Case Study of Updating Systematic Medical Reviews

“…The study evaluates the impact of imposing time constraints on clinicians in answering medical questions with the help of an SML search system. The study is still ongoing, and results from participants testing is not published yet [22].…”