Reference medical datasets (MosMedData) for independent external evaluation of algorithms based on artificial intelligence in diagnostics

Pavlov, Nikolai; Andreychenko, Anna; Vladzymyrskyy, A.V.; Revazyan, Anush A.; Kirpichev, Yury S.; Morozov, S. P.

doi:10.17816/dd60635

Cited by 18 publications

(9 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…a method for establishing the required accuracy for estimated sensitivity and specificity claimed by the developer) [ 27 , 28 ]. The data set building process is presented in detail in the papers [ 29 - 31 ].…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Methodology for Conducting Post-Marketing Surveillance of Software as a Medical Device Based on Artificial Intelligence Technologies

Zinchenko¹,

Arzamasov²,

Chetverikov³

et al. 2022

Sovrem Tehnol Med

View full text Add to dashboard Cite

The aim of the study was to develop a methodology for conducting post-registration clinical monitoring of software as a medical device based on artificial intelligence technologies (SaMD-AI).Materials and Methods. The methodology of post-registration clinical monitoring is based on the requirements of regulatory legal acts issued by the Board of the Eurasian Economic Commission. To comply with these requirements, the monitoring involves submission of the review of adverse events reports, the review of developers' routine reports on the safety and efficiency of SaMD-AI, and the assessment of the system for collecting and analyzing developers' post-registration data on the safety and efficiency of medical devices. The methodology was developed with regard to the recommendations of the International Medical Device Regulators Forum and the documents issued by the Food and Drug Administration (USA). Field-testing of this methodology was carried out using SaMD-AI designed for diagnostic imaging.Results. The post-registration monitoring of SaMD-AI consists of three key stages: collecting user feedback, technical monitoring and clinical validation. Technical monitoring involves routine evaluation of SaMD-AI output data quality to detect and remove flaws in a timely manner, and to secure the product stability. Major outcomes include an ordered list of technical flaws in SaMD-AI and their classification using evidence from diagnostic imaging studies. The application of this methodology resulted in a gradual reduction in the number of studies with flaws due to timely improvements in artificial intelligence algorithms: the number of flaws decreased to 5% in various aspects during subsequent testing. Clinical validation confirmed that SaMD-AI is capable of producing clinically meaningful outputs related to its intended use within the functionality determined by the developer. The testing procedure and the baseline testing framework were established during the field testing. Conclusion.The developed methodology will ensure the safety and efficiency of SaMD-AI taking into account its specifics as intangible medical devices. The methodology presented in this paper can be used by SaMD-AI developers to plan and carry out the post-registration clinical monitoring.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Annotation types are presented in the papers [29,30]. The first stage of clinical validation is the formation of a verified data set, i.e., a set with confirmed clinical data.…”

Section: A T a L A B E L L I N G ( A N N O T A T I O N)mentioning

confidence: 99%

Methodology for Conducting Post-Marketing Surveillance of Software as a Medical Device Based on Artificial Intelligence Technologies

Zinchenko¹,

Arzamasov²,

Chetverikov³

et al. 2022

Sovrem Tehnol Med

View full text Add to dashboard Cite

show abstract

“…Appendix B presents the types of SaMD-AI changes in details with some examples. Major changes are divided into three types; they imply a subsequent change of a SaMD-AI version [19]:…”

Section: Methodsmentioning

confidence: 99%

Changes in software as a medical device based on artificial intelligence technologies

Zinchenko¹,

Chetverikov²,

Ахмад³

et al. 2022

Int J CARS

Self Cite

View full text Add to dashboard Cite

Purpose to develop a procedure for registering changes, notifying users about changes made, unifying software as a medical device based on artificial intelligence technologies (SaMD-AI) changes, as well as requirements for testing and inspectionsquality control before and after making changes. Methods The main types of changes, divided into two groups-major and minor. Major changes imply a subsequent change of a SaMD-AI version to improve efficiency and safety, to change the functionality, and to ensure the processing of new data types. Minor changes imply those that SaMD-AI developers can make due to errors in the program code. Three types of SaMD-AI testings are proposed to use: functional testing, calibration testing or control, and technical testing. ResultsThe presented approaches for validation SaMD-AI changes were introduced. The unified requirements for the request for changes and forms of their submission made this procedure understandable for SaMD-AI developers, and also adjusted the workload for the Experiment experts who checked all the changes made to SaMD-AI. Conclusion This article discusses the need to control changes in the module of SaMD-AI, as innovative products influencing medical decision making. It justifies the need to control a module operation of SaMD-AI after making changes. To streamline and optimize the necessary and sufficient control procedures, a systematization of possible changes in SaMD-AI and testing methods was carried out. KeywordsArtificial intelligence • Medical software based on artificial intelligence technologies • Software as a medical device • Modifications • Changes • Validation B Ekaterina Akhmad

show abstract

“…In addition to testing the models on public datasets, we also evaluated them on an independent test dataset collected by the Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department. The dataset was prepared to test various algorithms using the principles described in (Pavlov et al, 2021) and consists of 50 studies with vertebral fractures and 50 age-matched healthy studies (Genant score is 0.75 or higher). Cases were prepared carefully to test the algorithms under various conditions, such as vertebral ankylosis, vertebroplasty and osteoblastic metastases, among others.…”

Section: Mosmedai Test Datasetmentioning

confidence: 99%

“…First, we transfer it from LungCancer-500 to VerSe with a negligible drop in quality. Second, we tested the proposed approach within the mosmed.ai initiative, an experiment for systematic comparison of various AI solutions for medical imaging (Pavlov et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Interpretable Vertebral Fracture Quantification via Anchor-Free Landmarks Localization

Zakharov¹,

Pisov²,

Alim³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vertebral body compression fractures are early signs of osteoporosis. Though these fractures are visible on Computed Tomography (CT) images, they are frequently missed by radiologists in clinical settings. Prior research on automatic methods of vertebral fracture classification proves its reliable quality; however, existing methods provide hard-to-interpret outputs and sometimes fail to process cases with severe abnormalities such as highly pathological vertebrae or scoliosis. We propose a new two-step algorithm to localize the vertebral column in 3D CT images and then detect individual vertebrae and quantify fractures in 2D simultaneously. We train neural networks for both steps using a simple 6keypoints based annotation scheme, which corresponds precisely to the current clinical recommendation. Our algorithm has no exclusion criteria, processes 3D CT in 2 seconds on a single GPU, and provides an interpretable and verifiable output. The method approaches expert-level performance and demonstrates state-of-the-art results in vertebrae 3D localization (the average error is 1 mm), vertebrae 2D detection (precision and recall are 0.99), and fracture identification (ROC AUC at the patient level is up to 0.96). Our anchor-free vertebra detection network shows excellent generalizability on a new domain by achieving ROC AUC 0.95, sensitivity 0.85, specificity 0.9 on a challenging VerSe dataset with many unseen vertebra types .

show abstract

Reference medical datasets (MosMedData) for independent external evaluation of algorithms based on artificial intelligence in diagnostics

Cited by 18 publications

References 13 publications

Methodology for Conducting Post-Marketing Surveillance of Software as a Medical Device Based on Artificial Intelligence Technologies

Methodology for Conducting Post-Marketing Surveillance of Software as a Medical Device Based on Artificial Intelligence Technologies

Changes in software as a medical device based on artificial intelligence technologies

Interpretable Vertebral Fracture Quantification via Anchor-Free Landmarks Localization

Contact Info

Product

Resources

About