Enhancing trust in clinical decision support systems: a framework for developers

Jones, Caroline; Thornton, James R.; Wyatt, Jeremy

doi:10.1136/bmjhci-2020-100247

Cited by 21 publications

(12 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several factors have been reported to be associated with the uptake of CDSS applications. Jones et al [27] stated that the uptake of CDSS depends on trustworthiness of the application. The authors [27] argued that the developers should ensure that the mechanisms and the sources of information that the system follows to come to a conclusion are transparent to the users.…”

Section: Discussionmentioning

confidence: 99%

Evaluating Pilot Implementation of ‘PenCS Flu Topbar’ App in Medical Practices to Improve National Immunisation Program–Funded Seasonal Influenza Vaccination in Central Queensland, Australia

Khandaker,

Chapman,

Khan

et al. 2024

Influenza Resp Viruses

View full text Add to dashboard Cite

BackgroundThe ‘PenCS Flu Topbar’ app was deployed in Central Queensland (CQ), Australia, medical practices through a pilot programme in March 2021.MethodsWe evaluated the app's user experience and examined whether the introduction of ‘PenCS Flu Topbar’ in medical practices could improve the coverage of NIP‐funded influenza vaccinations. We conducted a mixed‐method study including a qualitative analysis of in‐depth interviews with key end‐users and a quantitative analysis of influenza vaccine administrative data.Results‘PenCS Flu Topbar’ app users reported positive experiences identifying patients eligible for NIP‐funded seasonal influenza vaccination. A total of 3606 NIP‐funded influenza vaccinations was administered in the eight intervention practices, 14% higher than the eight control practices. NIP‐funded vaccination coverage within practices was significantly higher in the intervention practices (31.2%) than in the control practices (27.3%) (absolute difference: 3.9%; 95% CI: 2.9%–5.0%; p < 0.001). The coverage was substantially higher in Aboriginal and Torres Strait Islander people aged more than 6 months, pregnant women and children aged 6 months to less than 5 years for the practices where the app was introduced when compared to control practices: incidence rate ratio (IRR) 2.4 (95% CI: 1.8–3.2), IRR 2.7 (95% CI: 1.8–4.2) and IRR 2.3 (1.8–2.9) times higher, respectively.ConclusionsOur evaluation indicated that the ‘PenCS Flu Topbar’ app is useful for identifying the patients eligible for NIP‐funded influenza vaccination and is likely to increase NIP‐funded influenza vaccine coverage in the eligible populations. Future impact evaluation including a greater number of practices and a wider geographical area is essential.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating Pilot Implementation of ‘PenCS Flu Topbar’ App in Medical Practices to Improve National Immunisation Program–Funded Seasonal Influenza Vaccination in Central Queensland, Australia

Khandaker,

Chapman,

Khan

et al. 2024

Influenza Resp Viruses

View full text Add to dashboard Cite

show abstract

“…This deliberative multi‐disciplinary process has generated several useful insights, such as the analogy with IVDs, and has allowed us to make the following conclusions: Regulators are unlikely to consider the biomedical knowledge stored in MCBK libraries as a medical device and regulate it, even if it relates to diagnosis and treatment, unless it takes the form of Boxwala et al's “executable knowledge,” that is, software, for example, a CDSS knowledge object that responds directly to a CDS Hooks request via an HL7 FHIR API 31 Describing an object in an MCBK digital knowledge library as “suitable for medical decision support” or “will improve patient care” is unwise; instead, such knowledge objects should be presented as pluripotent, that is, providing knowledge to support a wide range of non‐clinical as well as clinical tasks, such as using technology‐agnostic logic to derive compliance metrics. While most objects in MCBK libraries do not seem likely to be regulated, it is important to build professional and wider trust in these libraries and their contents 34 . This means adopting a standard approach to governance and quality management, perhaps through a library certification process.…”

Section: Discussionmentioning

confidence: 99%

Which computable biomedical knowledge objects will be regulated? Results of a UK workshop discussing the regulation of knowledge libraries and software as a medical device

Wyatt,

Scott,

Ordish

et al. 2023

Learning Health Systems

Self Cite

View full text Add to dashboard Cite

IntroductionTo understand when knowledge objects in a computable biomedical knowledge library are likely to be subject to regulation as a medical device in the United Kingdom.MethodsA briefing paper was circulated to a multi‐disciplinary group of 25 including regulators, lawyers and others with insights into device regulation. A 1‐day workshop was convened to discuss questions relating to our aim. A discussion paper was drafted by lead authors and circulated to other authors for their comments and contributions.ResultsThis article reports on those deliberations and describes how UK device regulators are likely to treat the different kinds of knowledge objects that may be stored in computable biomedical knowledge libraries. While our focus is the likely approach of UK regulators, our analogies and analysis will also be relevant to the approaches taken by regulators elsewhere. We include a table examining the implications for each of the four knowledge levels described by Boxwala in 2011 and propose an additional level.ConclusionsIf a knowledge object is described as directly executable for a medical purpose to provide decision support, it will generally be in scope of UK regulation as “software as a medical device.” However, if the knowledge object consists of an algorithm, a ruleset, pseudocode or some other representation that is not directly executable and whose developers make no claim that it can be used for a medical purpose, it is not likely to be subject to regulation. We expect similar reasoning to be applied by regulators in other countries.

show abstract

“…However, this introduction of novel risk into the consultation deserves a sound evidence base which is currently missing. Mistrust of decision support systems is a recognised issue59 and may be justified: there is limited evidence supporting eRATs in terms of improving survival and patient experience. 12,13 The role of clinical judgement in the mistrust of eRATs is a tension which has been described in other areas of evidence-based medicine 60 61.…”

Section: Discussionmentioning

confidence: 99%

How do electronic risk assessment tools affect the communication and understanding of diagnostic uncertainty in the primary care consultation? A systematic review and thematic synthesis

et al. 2022

View full text Add to dashboard Cite

ObjectivesTo conduct a systematic review and synthesise qualitative research of electronic risk assessment tools (eRATs) in primary care, examining how they affect the communication and understanding of diagnostic risk and uncertainty. eRATs are computer-based algorithms designed to help clinicians avoid missing important diagnoses, pick up possible symptoms early and facilitate shared decision-making.DesignSystematic search, using predefined criteria of the published literature and synthesis of the qualitative data, using Thematic Synthesis. Database searches on 27 November 2019 were of MEDLINE, Embase, CINAHL and Web of Science, and a secondary search of the references of included articles. Included studies were those involving electronic risk assessment or decision support, pertaining to diagnosis in primary care, where qualitative data were presented. Non-empirical studies and non-English language studies were excluded. 5971 unique studies were identified of which 441 underwent full-text review. 26 studies were included for data extraction. A further two were found from citation searches. Quality appraisal was via the CASP (Critical Appraisal Skills Program) tool. Data extraction was via line by line coding. A thematic synthesis was performed.SettingPrimary care.ResultseRATs included differential diagnosis suggestion tools, tools which produce a future risk of disease development or recurrence or calculate a risk of current undiagnosed disease. Analytical themes were developed to describe separate aspects of the clinical consultation where risk and uncertainty are both central and altered via the use of an eRAT: ‘Novel risk’, ‘Risk refinement’, ‘Autonomy’, ‘Communication’, ‘Fear’ and ‘Mistrust’.ConclusioneRATs may improve the understanding and communication of risk in the primary care consultation. The themes of ‘Fear’ and ‘Mistrust’ could represent potential challenges with eRATs.Trial registration numberCRD219446.

show abstract

Enhancing trust in clinical decision support systems: a framework for developers

Cited by 21 publications

References 30 publications

Evaluating Pilot Implementation of ‘PenCS Flu Topbar’ App in Medical Practices to Improve National Immunisation Program–Funded Seasonal Influenza Vaccination in Central Queensland, Australia

Evaluating Pilot Implementation of ‘PenCS Flu Topbar’ App in Medical Practices to Improve National Immunisation Program–Funded Seasonal Influenza Vaccination in Central Queensland, Australia

Which computable biomedical knowledge objects will be regulated? Results of a UK workshop discussing the regulation of knowledge libraries and software as a medical device

How do electronic risk assessment tools affect the communication and understanding of diagnostic uncertainty in the primary care consultation? A systematic review and thematic synthesis

Contact Info

Product

Resources

About