Building an evidence standards framework for artificial intelligence-enabled digital health technologies

Unsworth, Harriet; Wolfram, Verena; Dillon, B.M.; Salmon, Mark; Greaves, Felix; Liu, Xiaoxuan; MacDonald, Trystan; Denniston, Alastair K; Sounderajah, Viknesh; Ashrafian, Hutan; Darzi, Ara; Ashurst, Carolyn; Holmes, Chris; Weller, Adrian

doi:10.1016/s2589-7500(22)00030-9

Cited by 20 publications

(7 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The comprehensive PECUNIA Project represents a network of partners that aim to establish standardized costing and outcome assessment measures with a multi-sectorial and multi-national approach [ 58 ]. In the digital health arena, there are efforts underway to define DHIs [ 59 ], classify them according to risks [ 60 ], or build frameworks to assess their value [ 61 ]. As indicated by our scoping review, the question of DHI program costs has been raised from various perspectives but comprehensive guidance is lacking [ 62 ].…”

Section: Discussionmentioning

confidence: 99%

Developing a Program Costs Checklist of Digital Health Interventions: A Scoping Review and Empirical Case Study

Khan,

Kidholm,

Pedersen

et al. 2024

PharmacoEconomics

View full text Add to dashboard Cite

Introduction The rate of development and complexity of digital health interventions (DHIs) in recent years has to some extent outpaced the methodological development in economic evaluation and costing. Particularly, the choice of cost components included in intervention or program costs of DHIs have received scant attention. The aim of this study was to build a literature-informed checklist of program cost components of DHIs. The checklist was next tested by applying it to an empirical case, Mamma Mia, a DHI developed to prevent perinatal depression. Method A scoping review with a structured literature search identified peer-reviewed literature from 2010 to 2022 that offers guidance on program costs of DHIs. Relevant guidance was summarized and extracted elements were organized into categories of main cost components and their associated activities following the standard three-step approach, that is, activities, resource use and unit costs. Results Of the 3448 records reviewed, 12 studies met the criteria for data extraction. The main cost categories identified were development, research, maintenance, implementation and health personnel involvement (HPI). Costs are largely considered to be context-specific, may decrease as the DHI matures and vary with number of users. The five categories and their associated activities constitute the checklist. This was applied to estimate program costs per user for Mamma Mia Self-Guided and Blended, the latter including additional guidance from public health nurses during standard maternal check-ups. Excluding research, the program cost per mother was more than double for Blended compared with Self-Guided (€140.5 versus €56.6, 2022 Euros) due to increased implementation and HPI costs. Including research increased the program costs to €190.8 and €106.9, respectively. One-way sensitivity analyses showed sensitivity to changes in number of users, lifespan of the app, salaries and license fee. Conclusion The checklist can help increase transparency of cost calculation and improve future comparison across studies. Supplementary Information The online version contains supplementary material available at 10.1007/s40273-024-01366-y.

show abstract

Section: Discussionmentioning

confidence: 99%

Developing a Program Costs Checklist of Digital Health Interventions: A Scoping Review and Empirical Case Study

Khan,

Kidholm,

Pedersen

et al. 2024

PharmacoEconomics

View full text Add to dashboard Cite

show abstract

“…To generate AI-specific characteristics, we will therefore adapt standards from NICE’s ESF for digital health technologies, a gold standard tool designed to evaluate digital health technologies for UK commissioning decisions. This was revised in 2022 to encompass AI devices, with the new AI standards being generated through a multistakeholder Delphi consensus process [ 49 , 50 ]. The 2 reviewers (TM and JH) will review the ESF, extracting relevant standards, with disagreements resolved by discussion or arbitration by the senior investigator (AKD).…”

Section: Methodsmentioning

confidence: 99%

Target Product Profile for a Machine Learning–Automated Retinal Imaging Analysis Software for Use in English Diabetic Eye Screening: Protocol for a Mixed Methods Study

Macdonald,

Dinnes,

Maniatopoulos

et al. 2024

JMIR Res Protoc

Self Cite

View full text Add to dashboard Cite

Background Diabetic eye screening (DES) represents a significant opportunity for the application of machine learning (ML) technologies, which may improve clinical and service outcomes. However, successful integration of ML into DES requires careful product development, evaluation, and implementation. Target product profiles (TPPs) summarize the requirements necessary for successful implementation so these can guide product development and evaluation. Objective This study aims to produce a TPP for an ML-automated retinal imaging analysis software (ML-ARIAS) system for use in DES in England. Methods This work will consist of 3 phases. Phase 1 will establish the characteristics to be addressed in the TPP. A list of candidate characteristics will be generated from the following sources: an overview of systematic reviews of diagnostic test TPPs; a systematic review of digital health TPPs; and the National Institute for Health and Care Excellence’s Evidence Standards Framework for Digital Health Technologies. The list of characteristics will be refined and validated by a study advisory group (SAG) made up of representatives from key stakeholders in DES. This includes people with diabetes; health care professionals; health care managers and leaders; and regulators and policy makers. In phase 2, specifications for these characteristics will be drafted following a series of semistructured interviews with participants from these stakeholder groups. Data collected from these interviews will be analyzed using the shortlist of characteristics as a framework, after which specifications will be drafted to create a draft TPP. Following approval by the SAG, in phase 3, the draft will enter an internet-based Delphi consensus study with participants sought from the groups previously identified, as well as ML-ARIAS developers, to ensure feasibility. Participants will be invited to score characteristic and specification pairs on a scale from “definitely exclude” to “definitely include,” and suggest edits. The document will be iterated between rounds based on participants’ feedback. Feedback on the draft document will be sought from a group of ML-ARIAS developers before its final contents are agreed upon in an in-person consensus meeting. At this meeting, representatives from the stakeholder groups previously identified (minus ML-ARIAS developers, to avoid bias) will be presented with the Delphi results and feedback of the user group and asked to agree on the final contents by vote. Results Phase 1 was completed in November 2023. Phase 2 is underway and expected to finish in March 2024. Phase 3 is expected to be complete in July 2024. Conclusions The multistakeholder development of a TPP for an ML-ARIAS for use in DES in England will help developers produce tools that serve the needs of patients, health care providers, and their staff. The TPP development process will also provide methods and a template to produce similar documents in other disease areas. International Registered Report Identifier (IRRID) DERR1-10.2196/50568

show abstract

“…Another strength of the UK approach thus far has been to begin the integration of the 2 separate state oversight systems for AIeMD market access, that is, the regulatory layer and the health technology assessment layer. There is interaction and common principles between the proposal for the new UK MDR [ 9 ], and NICE (the UK health technology assessment body) recently proposed “Evidence standards framework (ESF) for digital health technologies,” including for measuring changes in the performance of DHTs over time [ 56 , 57 ].…”

Section: Challenges and Opportunitiesmentioning

confidence: 99%

Learning From Experience and Finding the Right Balance in the Governance of Artificial Intelligence and Digital Health Technologies

Gilbert¹,

Anderson²,

Daumer³

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

Artificial intelligence (AI) and machine learning medical tools have the potential to be transformative in care delivery; however, this change will only be realized if accompanied by effective governance that ensures patient safety and public trust. Recent digital health initiatives have called for tighter governance of digital health. A correct balance must be found between ensuring product safety and performance while also enabling the innovation needed to deliver better approaches for patients and affordable efficient health care for society. This requires innovative, fit-for-purpose approaches to regulation. Digital health technologies, particularly AI-based tools, pose specific challenges to the development and implementation of functional regulation. The approaches of regulatory science and “better regulation” have a critical role in developing and evaluating solutions to these problems and ensuring effective implementation. We describe the divergent approaches of the European Union and the United States in the implementation of new regulatory approaches in digital health, and we consider the United Kingdom as a third example, which is in a unique position of developing a new post-Brexit regulatory framework.

show abstract

Building an evidence standards framework for artificial intelligence-enabled digital health technologies

Cited by 20 publications

References 4 publications

Developing a Program Costs Checklist of Digital Health Interventions: A Scoping Review and Empirical Case Study

Developing a Program Costs Checklist of Digital Health Interventions: A Scoping Review and Empirical Case Study

Target Product Profile for a Machine Learning–Automated Retinal Imaging Analysis Software for Use in English Diabetic Eye Screening: Protocol for a Mixed Methods Study

Learning From Experience and Finding the Right Balance in the Governance of Artificial Intelligence and Digital Health Technologies

Contact Info

Product

Resources

About