Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products

Wang, Lu; Wang, Hao; Chen, Xia; Wang, Yao; Tang, Qiaohong; Li, Jiage; Zhou, Xiao‐Hua

doi:10.1080/17434440.2020.1813566

Cited by 8 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an emerging branch of the medical device, the AIMD, along with increasing applications of deep learning [1,2], has demonstrated signifcant potential in medical imaging, image reconstruction, and postprocessing [3][4][5][6][7][8][9][10][11][12][13][14][15][16]. While hundreds of AIMDs have been approved [17,18], the verifcation and validation of such devices are mainly conducted by manufacturers spontaneously, leading to variation in evaluation metrics and data sets [19]. Stakeholders show rising concern on the quality of the AIMD, such as its comparability [20] and transparency [21], which poses considerable challenges to regulation compared to a conventional medical device.…”

Section: Introductionmentioning

confidence: 99%

Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer‐Aided Diagnosis of Diabetic Retinopathy

Wang

Meng

Tang

et al. 2023

Journal of Healthcare Engineering

Self Cite

View full text Add to dashboard Cite

Objective. To explore a centralized approach to build test sets and assess the performance of an artificial intelligence medical device (AIMD) which is intended for computer-aided diagnosis of diabetic retinopathy (DR). Method. A framework was proposed to conduct data collection, data curation, and annotation. Deidentified colour fundus photographs were collected from 11 partner hospitals with raw labels. Photographs with sensitive information or authenticity issues were excluded during vetting. A team of annotators was recruited through qualification examinations and trained. The annotation process included three steps: initial annotation, review, and arbitration. The annotated data then composed a standardized test set, which was further imported to algorithms under test (AUT) from different developers. The algorithm outputs were compared with the final annotation results (reference standard). Result. The test set consists of 6327 digital colour fundus photographs. The final labels include 5 stages of DR and non-DR, as well as other ocular diseases and photographs with unacceptable quality. The Fleiss Kappa was 0.75 among the annotators. The Cohen’s kappa between raw labels and final labels is 0.5. Using this test set, five AUTs were tested and compared quantitatively. The metrics include accuracy, sensitivity, and specificity. The AUTs showed inhomogeneous capabilities to classify different types of fundus photographs. Conclusions. This article demonstrated a workflow to build standardized test sets and conduct algorithm testing of the AIMD for computer-aided diagnosis of diabetic retinopathy. It may provide a reference to develop technical standards that promote product verification and quality control, improving the comparability of products.

show abstract

Section: Introductionmentioning

confidence: 99%

Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer‐Aided Diagnosis of Diabetic Retinopathy

Wang

Meng

Tang

et al. 2023

Journal of Healthcare Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…A large BrixlA dataset of CXR images related to COVID-19 are used as the training, validation, and test data. The AUC, sensitivity, and specificity are utilized as the performance evaluation items because they are mainly evaluated through FDA premarket approvals [ 28 ]. The major outcomes of this study are the following: 1) All performances were rapidly improved as the number of training data were increased and reached an equilibrium state.…”

Section: Introductionmentioning

confidence: 99%

Performance change with the number of training data: A case study on the binary classification of COVID-19 chest X-ray by using convolutional neural networks

Imagawa¹,

Shiomoto²

2022

Computers in Biology and Medicine

View full text Add to dashboard Cite

“…While stakeholders are interested to compare products and understand their quality, the verification and validation of such products is often conducted by manufacturers individually. Currently, there are differences in the performance metrics and verification methods claimed by different manufacturers, resulting in a lack of comparability between algorithms ( 41 ). There is also a lack of understanding of the common quality characteristics of these algorithms.…”

Section: Introductionmentioning

confidence: 99%

Practice toward standardized performance testing of computer-aided detection algorithms for pulmonary nodule

Wang

Tang

Zhang

et al. 2022

Front. Public Health

Self Cite

View full text Add to dashboard Cite

This study aimed at implementing practice to build a standardized protocol to test the performance of computer-aided detection (CAD) algorithms for pulmonary nodules. A test dataset was established according to a standardized procedure, including data collection, curation and annotation. Six types of pulmonary nodules were manually annotated as reference standard. Three specific rules to match algorithm output with reference standard were applied and compared. These rules included: (1) “center hit” [whether the center of algorithm highlighted region of interest (ROI) hit the ROI of reference standard]; (2) “center distance” (whether the distance between algorithm highlighted ROI center and reference standard center was below a certain threshold); (3) “area overlap” (whether the overlap between algorithm highlighted ROI and reference standard was above a certain threshold). Performance metrics were calculated and the results were compared among ten algorithms under test (AUTs). The test set currently consisted of CT sequences from 593 patients. Under “center hit” rule, the average recall rate, average precision, and average F1 score of ten algorithms under test were 54.68, 38.19, and 42.39%, respectively. Correspondingly, the results under “center distance” rule were 55.43, 38.69, and 42.96%, and the results under “area overlap” rule were 40.35, 27.75, and 31.13%. Among the six types of pulmonary nodules, the AUTs showed the highest miss rate for pure ground-glass nodules, with an average of 59.32%, followed by pleural nodules and solid nodules, with an average of 49.80 and 42.21%, respectively. The algorithm testing results changed along with specific matching methods adopted in the testing process. The AUTs showed uneven performance on different types of pulmonary nodules. This centralized testing protocol supports the comparison between algorithms with similar intended use, and helps evaluate algorithm performance.

show abstract

Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products

Cited by 8 publications

References 22 publications

Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer‐Aided Diagnosis of Diabetic Retinopathy

Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer‐Aided Diagnosis of Diabetic Retinopathy

Performance change with the number of training data: A case study on the binary classification of COVID-19 chest X-ray by using convolutional neural networks

Practice toward standardized performance testing of computer-aided detection algorithms for pulmonary nodule

Contact Info

Product

Resources

About