Systematic analysis of the test design and performance of AI/ML-based medical devices approved for triage/detection/diagnosis in the USA and Japan

Yuba, Mitsuru; Iwasaki, Katsurô

doi:10.1038/s41598-022-21426-7

Cited by 11 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AI research in radiology is an active field in healthcare owing to its affinity for imaging [ 14 ], with the number of AI-related publications and medical device certifications increasing annually [ 15 – 17 ]. One important reason for this is the global shortage of radiologists [ 18 – 21 ].…”

Section: Introductionmentioning

confidence: 99%

Fairness of artificial intelligence in healthcare: review and recommendations

Ueda¹,

Kakinuma²,

Fujita

et al. 2023

Jpn J Radiol

116

View full text Add to dashboard Cite

In this review, we address the issue of fairness in the clinical integration of artificial intelligence (AI) in the medical field. As the clinical adoption of deep learning algorithms, a subfield of AI, progresses, concerns have arisen regarding the impact of AI biases and discrimination on patient health. This review aims to provide a comprehensive overview of concerns associated with AI fairness; discuss strategies to mitigate AI biases; and emphasize the need for cooperation among physicians, AI researchers, AI developers, policymakers, and patients to ensure equitable AI integration. First, we define and introduce the concept of fairness in AI applications in healthcare and radiology, emphasizing the benefits and challenges of incorporating AI into clinical practice. Next, we delve into concerns regarding fairness in healthcare, addressing the various causes of biases in AI and potential concerns such as misdiagnosis, unequal access to treatment, and ethical considerations. We then outline strategies for addressing fairness, such as the importance of diverse and representative data and algorithm audits. Additionally, we discuss ethical and legal considerations such as data privacy, responsibility, accountability, transparency, and explainability in AI. Finally, we present the Fairness of Artificial Intelligence Recommendations in healthcare (FAIR) statement to offer best practices. Through these efforts, we aim to provide a foundation for discussing the responsible and equitable implementation and deployment of AI in healthcare.

show abstract

Section: Introductionmentioning

confidence: 99%

Fairness of artificial intelligence in healthcare: review and recommendations

Ueda¹,

Kakinuma²,

Fujita

et al. 2023

Jpn J Radiol

116

View full text Add to dashboard Cite

show abstract

“…Artificial intelligence algorithms can assist human clinicians in various medical fields 4 , 5 . Among the artificial intelligence algorithms, the reinforcement learning algorithms can find the optimal policy by maximizing the cumulative expected reward 6 .…”

Section: Introductionmentioning

confidence: 99%

Development and validation of a reinforcement learning model for ventilation control during emergence from general anesthesia

et al. 2023

View full text Add to dashboard Cite

Ventilation should be assisted without asynchrony or cardiorespiratory instability during anesthesia emergence until sufficient spontaneous ventilation is recovered. In this multicenter cohort study, we develop and validate a reinforcement learning-based Artificial Intelligence model for Ventilation control during Emergence (AIVE) from general anesthesia. Ventilatory and hemodynamic parameters from 14,306 surgical cases at an academic hospital between 2016 and 2019 are used for training and internal testing of the model. The model’s performance is also evaluated on the external validation cohort, which includes 406 cases from another academic hospital in 2022. The estimated reward of the model’s policy is higher than that of the clinicians’ policy in the internal (0.185, the 95% lower bound for best AIVE policy vs. −0.406, the 95% upper bound for clinicians’ policy) and external validation (0.506, the 95% lower bound for best AIVE policy vs. 0.154, the 95% upper bound for clinicians’ policy). Cardiorespiratory instability is minimized as the clinicians’ ventilation matches the model’s ventilation. Regarding feature importance, airway pressure is the most critical factor for ventilation control. In conclusion, the AIVE model achieves higher estimated rewards with fewer complications than clinicians’ ventilation control policy during anesthesia emergence.

show abstract

“…A recent study on the systematic analysis to clarify the test design of AI/ML-based CAD at pre-market demonstrated that CAST was approved based on SA, whereas CADe and CADx were approved based on RT in the U.S. [14] However, no studies have been conducted to determine how post-market improvement should be evaluated. Therefore, the lack of predictability of whether post-market improvements should be evaluated using SA or RT is an obstacle for start-ups and other companies developing medical devices based on AI technology.…”

Section: Introductionmentioning

confidence: 99%

Performance evaluation methods for improvements at post-market of artificial intelligence/machine learning-based computer-aided detection/diagnosis/triage in the United States

Yuba

Iwasaki²

2023

PLOS Digit Health

Self Cite

View full text Add to dashboard Cite

Computer-aided detection (CADe), computer-aided diagnosis (CADx), and computer-aided simple triage (CAST), which incorporate artificial intelligence (AI) and machine learning (ML), are continually undergoing post-market improvement. Therefore, understanding the evaluation and approval process of improved products is important. This study intended to conduct a comprehensive survey of AI/ML-based CAD products approved by the U.S. Food and Drug Administration (FDA) that had been improved post-market to gain insights into the efficacy and safety required for market approval. A survey of the product code database published by the FDA identified eight products that were improved post-market. The methods used to evaluate the performance of improvements were analysed, and post-market improvements were approved with retrospective data. Reader study testing (RT) or software standalone testing (SA) procedures were conducted retrospectively. Six RT procedures were conducted because of modifications to the intended use. An average of 17.3 readers (minimum 14, maximum 24) participated, and the area under the curve (AUC) was considered the primary endpoint. The addition of study learning data that did not change the intended use and changes in the analysis algorithm were evaluated by SA. The average sensitivity, specificity, and AUC were 93% (minimum 91.1, maximum 97), 89.6% (minimum 85.9, maximum 96), and 0.96 (minimum 0.96, maximum 0.97), respectively. The average interval between applications was 348 days (minimum –18, maximum 975), which showed that the improvements were implemented within approximately one year. This is the first comprehensive study on AI/ML-based CAD products that have been improved post-market to elucidate evaluation points for post-market improvements. The findings will be informative for the industry and academia in developing and improving AI/ML-based CAD.

show abstract

Systematic analysis of the test design and performance of AI/ML-based medical devices approved for triage/detection/diagnosis in the USA and Japan

Cited by 11 publications

References 27 publications

Fairness of artificial intelligence in healthcare: review and recommendations

Fairness of artificial intelligence in healthcare: review and recommendations

Development and validation of a reinforcement learning model for ventilation control during emergence from general anesthesia

Performance evaluation methods for improvements at post-market of artificial intelligence/machine learning-based computer-aided detection/diagnosis/triage in the United States

Contact Info

Product

Resources

About