To buy or not to buy—evaluating commercial AI solutions in radiology (the ECLAIR guidelines)

Abstract

Artificial intelligence (AI) has made impressive progress over the past few years, including many applications in medical imaging. Numerous commercial solutions based on AI techniques are now available for sale, forcing radiology practices to learn how to properly assess these tools. While several guidelines describing good practices for conducting and reporting AI-based research in medicine and radiology have been published, fewer efforts have focused on recommendations addressing the key questions to consider when critically assessing AI solutions before purchase. Commercial AI solutions are typically complicated software products, for the evaluation of which many factors are to be considered. In this work, authors from academia and industry have joined efforts to propose a practical framework that will help stakeholders evaluate commercial AI solutions in radiology (the ECLAIR guidelines) and reach an informed decision. Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services. Key Points • Numerous commercial solutions based on artificial intelligence techniques are now available for sale, and radiology practices have to learn how to properly assess these tools. • We propose a framework focusing on practical points to consider when assessing an AI solution in medical imaging, allowing all stakeholders to conduct relevant discussions with manufacturers and reach an informed decision as to whether to purchase an AI commercial solution for imaging applications. • Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services.

Technical and clinical validation of commercial automated volumetric MRI tools for dementia diagnosis—a systematic review

“…In this paper, they outline plans to update current regulatory frameworks, strengthen the harmonized development of "good machine learning practice", support a patientcentred approach and, most relevant to this review, support the development of methods for evaluating and improving machine learning algorithms and promote real-world performance studies, in other words, technical and clinical validation. The ECLAIR guidelines were also published very recently aiming to provide guidance and informed decisionmaking when evaluating commercial AI solutions in radiology before purchase [106].…”

Evaluation framework to guide implementation of AI systems into healthcare settings

“…One of the major limitations with many existing reporting or evaluation frameworks is their narrow focus. Some focus on reporting of clinical trials evaluating AI interventions 6 7 on a specific medical domain 20 21 or compare a particular type of AI model to human clinicians 8 limiting the generalisability of such frameworks. It is now increasingly becoming evident that many AI systems, that have shown promise in in-silico environment or when deployed in single sites, are not fit for purpose when deployed widely.…”

AI MSK clinical applications: spine imaging

“…Therefore, recent guidelines have been proposed by Omoumi et al, aiming for a "voluntary" standardization in terms of technical, financial, quality, and safety aspects in the evaluation of AI medical products. The so-called ECLAIR guidelines are the first ones which aim to summarize all aspects affecting all relevant stakeholders, and which propose strategies for radiology practices regarding how to properly assess AI tools [66].…”